Project 2: Design a tool to optimize the parameters of any command line tool

[ttimbers]

Project: Given a command line tool with a number of parameters and a target metric to be optimized, I want a tool that will run the program and find the values for those parameters that maximizes some target metric. My particular use case is genome sequence assembly, which often has a variety of parameters related to expected coverage of the reads and heuristics to remove read errors and collapse heterozygous variation. When I tackle that optimization, the process is manual and tedious: submitting jobs to a scheduler, rerunning failed jobs, inspecting outputs, tweaking parameters, and repeating. I want to design and implement a tool to automate that process and generate a report of the result.

Project Lead: Shaun Jackman / @sjackman / Graduate Student / BC Cancer Agency Genome Sciences Centre

[hyounesy]

Interesting.
I have also done some work on general parameter analysis of R/bioconductor packages as part of the VisRseq framework that might be applicable for this problem. I would be happy to provide further info if this may be of immediate use for you.

[sjackman]

Hi, Hamid. Thanks for your interest. One approach I had considered was using R functions like optim and optimize. Which R optimization function did you use?

I had also considered implementing simple optimization methods like grid search, line search, twiddle and amoeba aka Nelder–Mead. However it's implemented, I want it to be usable as a command-line program.

[hyounesy]

Makes sense.
For VisRseq (which is not command-line) I am currently using a simple randomized search. I was interested in visual parameter exploration where a target metric does not necessarily exist. For example, for differential expression analysis tools (e.g. edgeR, DESeq, etc.), different parameters result in different genes detected as differentially expressed, but it is not always clear which result is better. I am however considering to experiment with better sampling (e.g. monte carlo) to distribute the parameter samples in the areas where there is more variation in the results.

[sjackman]

There are two existing tools for parameter optimization:

• Biopsy - the Bioinformatic Optimisation System by @blahah
• ParOpt - Generic command-line parameter optimization by @sseemayer

We definitely need to explore whether building on one of these existing tools fits our requirements.

There's also the possibility of building on other optimization systems such as R's optimize family of functions or Python's scipy.optimize. I don't know whether either of these handles optimization of discrete variables / metrics.

[sseemayer]

Sounds like a cool idea to have generic parameter optimization for cluster jobs 👍 !

ParOpt is a fairly simple script based on scipy.optimize that is limited to jobs running synchronously on one machine where the objective function to be optimized can be parsed from the command line's STDOUT. It does not support asynchronous execution or clusters at all (except for maybe jury rigging some shell scripts to submit/evaluate jobs).

Thinking about it, generic optimization could actually benefit greatly from parallelization. Since these generic methods cannot rely on gradients and are typically very slow to run, parallelizing the problem will make the wait times much shorter. Parallelization is definitely possible for grid search since that is embarrassingly parallel, but a quick google search gave me this article on parallel Nelder-Mead too.

It might be possible to run the function evaluations using DRMAA to abstract between different scheduling systems.

I'd be happy to answer any questions you might have about ParOpt 😄

[sjackman]

Great! Thanks for your response, Stefan! I'll likely have more questions for you. Any chance you'd like to come to Vancouver, Canada for Hackseq Oct 15–17 and/or ASHG 2016 following it? 😄 You too, Richard @blahah! Come join us! Three ASHG sponsored travel awards ($1,500) are available!

[sseemayer]

Both your hackathon and conference sound amazing and I could visit my sister that lives in Redmond, WA so your offer is doubly tempting. On the other hand, I'll have just started my postdoc around then so I'll pass this time. I might come back on that offer next year, though!

[sjackman]

😢 Best of luck with your postdoc!

[sjackman]

We're planning to have a Docker image with a bunch of bioinformatics software preinstalled running on machines at the BC Cancer Agency Genome Sciences Centre during the Hackathon. Which bioinformatics software do you plant to use for your project? In particular, is there any software that you plan to use that is not already listed here? http://www.bcgsc.ca/services/orca

[sjackman]

I'll be using the two tools listed above, Biopsy and ParOpt, as well as Python scipy and R.

Yes, I'm talking to myself. I'm leading a project and also helping to organize the hackathon. =)

[whitecat]

This project seems like a cool area of work, running and testing is always a big problem in programming.

What features would you want more than Biopsy and ParOpt already offers? @sseemayer seems to say adding parallelization for cluster jobs. @sjackman refers to adding a different algorithm using R.

[sjackman]

I need to set aside some time to test out Biopsy and ParOpt, and to write up the requirements for this project. Off the top of my head:

• Run jobs in parallel (on the local machine for sure, possibly on a cluster)
• Generate a report of the results using RMarkdown plotting the outcome vs parameters
• Possibly a live updating report (using e.g. RStudio Dashboard) showing the combinations of parameters tried so far, and the best outcome so far

Other suggestions welcome!