Thanks to visit codestin.com
Credit goes to www.nature.com

Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Toil enables reproducible, open source, big biomedical data analyses

Nature Biotechnology volume 35, pages 314–316 (2017)Cite this article

Subjects

Access through your institution
Buy or subscribe

To the Editor:

Contemporary genomic data sets contain tens of thousands of samples and petabytes of sequencing data1,2,3. Pipelines to process genomic data sets often comprise dozens of individual steps, each with their own set of parameters4,5. Processing data at this scale and complexity is expensive, can take an unacceptably long time, and requires significant engineering effort. Furthermore, biomedical data sets are often siloed, both for organizational and security considerations and because they are physically difficult to transfer between systems, owing to bandwidth limitations. The solution to better handling these big data problems is twofold: first, we need robust software capable of running analyses quickly and efficiently, and second, we need the software and pipelines to be portable, so that they can be reproduced in any suitable compute environment.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: RNA-seq pipeline and expression concordance.
Figure 2: Costs and core usage.

References

  1. Weinstein, J.N. et al. Nat. Genet. 45, 1113–1120 (2013).

    Article  Google Scholar 

  2. Zhang, J. et al. Database. http://dx.doi.org/10.1093/database/bar026 (2011)

  3. Siva, N. Lancet 385, 103–104 (2015).

    Article  Google Scholar 

  4. McKenna, A. et al. Genome Res. 20, 1297–1303 (2010).

    Article  CAS  Google Scholar 

  5. UNC Bioinformatics. TCGA mRNA-seq pipeline for UNC data. https://webshare.bioinf.unc.edu/public/mRNAseq_TCGA/UNC_mRNAseq_summary.pdf (2013).

  6. Albrecht, M., Michael, A., Patrick, D., Peter, B. & Douglas, T. in Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies (SWEET '12) 1. ACM (Association of Computing Machinery. http://dx.doi.org/10.1145/2443416.2443417 (2012).

    Google Scholar 

  7. Bernhardsson, E. & Frieder, E. Luigi. Github https://github.com/spotify/luigi (2016).

    Google Scholar 

  8. Goecks, J., Nekrutenko, A. & Taylor, J. Genome Biol. 11, R86 (2010).

    Article  Google Scholar 

  9. UCSC. Xena http://xena.ucsc.edu (2016).

  10. Amstutz, P. Common workflow language. Github https://github.com/common-workflow-language/common-workflow-language (2016).

    Google Scholar 

  11. Frazer, S. Workflow description language. Github https://github.com/broadinstitute/wdl (2014).

    Google Scholar 

  12. Vivian, J. Toil scripts. Github https://github.com/BD2KGenomics/toil-scripts/tree/master/src/toil_scripts (2016).

    Google Scholar 

  13. Apache Software Foundation. Apache Spark http://spark.apache.org/ (2017).

  14. Massie, M. et al. ADAM: genomics formats and processing patterns for cloud scale computing. University of California, Berkeley, Technical Report No. UCB/EECS-2013-207 (2013).

  15. Gentzsch, W. in Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid 35–36 http://dx.doi.org/10.1109/ccgrid.2001.923173 (IEEE, 2001).

    Book  Google Scholar 

  16. Yoo, A.B., Jette, M.A. & Mark, G. in Lecture Notes in Computer Science 44–60 (2003) Springer, Berlin, Heidelberg.

    Google Scholar 

  17. Apache Software Foundation. Apache Mesos http://mesos.apache.org/

  18. GTEx Consortium. Science 348, 648–660 (2015).

  19. Dobin, A. et al. Bioinformatics 29, 15–21 (2013).

    Article  CAS  Google Scholar 

  20. Li, B. & Dewey, C.N. BMC Bioinformatics 12, 323 (2011).

    Article  CAS  Google Scholar 

  21. Bray, N.L., Pimentel, H., Melsted, P. & Pachter, L. Nat. Biotechnol. 34, 525–527 (2016).

    Article  CAS  Google Scholar 

  22. Barker, A.D. et al. Clin. Pharmacol. Ther. 86, 97–100 (2009).

    Article  CAS  Google Scholar 

  23. Kent, W.J. et al. Genome Res. 12, 996–1006 (2002).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

This work was supported by (BD2K) the National Human Genome Research Institute of the National Institutes of Health award no. 5U54HG007990 and (Cloud Pilot) the National Cancer Institute of the National Institutes of Health under the Broad Institute subaward no. 5417071-5500000716. The UCSC Genome Browser work was supported by the NHGRI award 5U41HG002371 (Corporate Sponsors). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or our corporate sponsors.

Author information

Authors and Affiliations

  1. Computational Genomics Lab, UC Santa Cruz Genomics Institute, University of California Santa Cruz, Santa Cruz, California, USA

    John Vivian, Arjun Arkal Rao, Christopher Ketchum, Joel Armstrong, Adam Novak, Jacob Pfeil, Jake Narkizian, Alden D Deran, Audrey Musselman-Brown, Hannes Schmidt, Brian Craft, Mary Goldman, Kate Rosenbloom, Melissa Cline, Brian O'Connor, W James Kent, Jingchun Zhu, David Haussler & Benedict Paten

  2. AMP Lab, University of California Berkeley, Berkeley, California, USA

    Frank Austin Nothaft, David A Patterson & Anthony D Joseph

  3. UC Berkeley ASPIRE Lab, Berkeley, Berkeley, California, USA

    Frank Austin Nothaft, David A Patterson & Anthony D Joseph

  4. Curoverse, Somerville, Massachusetts, USA

    Peter Amstutz & Sasha Zaranek

  5. Broad Institute of Harvard and Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, USA

    Megan Hanna, Chet Birger & Gad Getz

Authors
  1. John Vivian
    View author publications

    Search author on:PubMed Google Scholar

  2. Arjun Arkal Rao
    View author publications

    Search author on:PubMed Google Scholar

  3. Frank Austin Nothaft
    View author publications

    Search author on:PubMed Google Scholar

  4. Christopher Ketchum
    View author publications

    Search author on:PubMed Google Scholar

  5. Joel Armstrong
    View author publications

    Search author on:PubMed Google Scholar

  6. Adam Novak
    View author publications

    Search author on:PubMed Google Scholar

  7. Jacob Pfeil
    View author publications

    Search author on:PubMed Google Scholar

  8. Jake Narkizian
    View author publications

    Search author on:PubMed Google Scholar

  9. Alden D Deran
    View author publications

    Search author on:PubMed Google Scholar

  10. Audrey Musselman-Brown
    View author publications

    Search author on:PubMed Google Scholar

  11. Hannes Schmidt
    View author publications

    Search author on:PubMed Google Scholar

  12. Peter Amstutz
    View author publications

    Search author on:PubMed Google Scholar

  13. Brian Craft
    View author publications

    Search author on:PubMed Google Scholar

  14. Mary Goldman
    View author publications

    Search author on:PubMed Google Scholar

  15. Kate Rosenbloom
    View author publications

    Search author on:PubMed Google Scholar

  16. Melissa Cline
    View author publications

    Search author on:PubMed Google Scholar

  17. Brian O'Connor
    View author publications

    Search author on:PubMed Google Scholar

  18. Megan Hanna
    View author publications

    Search author on:PubMed Google Scholar

  19. Chet Birger
    View author publications

    Search author on:PubMed Google Scholar

  20. W James Kent
    View author publications

    Search author on:PubMed Google Scholar

  21. David A Patterson
    View author publications

    Search author on:PubMed Google Scholar

  22. Anthony D Joseph
    View author publications

    Search author on:PubMed Google Scholar

  23. Jingchun Zhu
    View author publications

    Search author on:PubMed Google Scholar

  24. Sasha Zaranek
    View author publications

    Search author on:PubMed Google Scholar

  25. Gad Getz
    View author publications

    Search author on:PubMed Google Scholar

  26. David Haussler
    View author publications

    Search author on:PubMed Google Scholar

  27. Benedict Paten
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Benedict Paten.

Ethics declarations

Competing interests

The authors received support from AWS, Microsoft, and Google.

Supplementary information

Supplementary Figures and Texts

Supplementary Notes 1–9, Supplementary Figures 1–4, Supplementary Tables 1 (PDF 2647 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vivian, J., Rao, A., Nothaft, F. et al. Toil enables reproducible, open source, big biomedical data analyses. Nat Biotechnol 35, 314–316 (2017). https://doi.org/10.1038/nbt.3772

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nbt.3772

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing