Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Research output: Contribution to journal › Journal article › Research › peer-review

Standard

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks. / Koehler Leman, Julia; Lyskov, Sergey; Lewis, Steven M.; Adolf-Bryfogle, Jared; Alford, Rebecca F.; Barlow, Kyle; Ben-Aharon, Ziv; Farrell, Daniel; Fell, Jason; Hansen, William A.; Harmalkar, Ameya; Jeliazkov, Jeliazko; Kuenze, Georg; Krys, Justyna D.; Ljubetič, Ajasja; Loshbaugh, Amanda L.; Maguire, Jack; Moretti, Rocco; Mulligan, Vikram Khipple; Nance, Morgan L.; Nguyen, Phuong T.; Ó Conchúir, Shane; Roy Burman, Shourya S.; Samanta, Rituparna; Smith, Shannon T.; Teets, Frank; Tiemann, Johanna K. S.; Watkins, Andrew; Woods, Hope; Yachnin, Brahm J.; Bahl, Christopher D.; Bailey-Kellogg, Chris; Baker, David; Das, Rhiju; DiMaio, Frank; Khare, Sagar D.; Kortemme, Tanja; Labonte, Jason W.; Lindorff-Larsen, Kresten; Meiler, Jens; Schief, William; Schueler-Furman, Ora; Siegel, Justin B.; Stein, Amelie; Yarov-Yarovoy, Vladimir; Kuhlman, Brian; Leaver-Fay, Andrew; Gront, Dominik; Gray, Jeffrey J.; Bonneau, Richard.

In: Nature Communications, Vol. 12, 6947, 2021.

Research output: Contribution to journal › Journal article › Research › peer-review

Harvard

Koehler Leman, J, Lyskov, S, Lewis, SM, Adolf-Bryfogle, J, Alford, RF, Barlow, K, Ben-Aharon, Z, Farrell, D, Fell, J, Hansen, WA, Harmalkar, A, Jeliazkov, J, Kuenze, G, Krys, JD, Ljubetič, A, Loshbaugh, AL, Maguire, J, Moretti, R, Mulligan, VK, Nance, ML, Nguyen, PT, Ó Conchúir, S, Roy Burman, SS, Samanta, R, Smith, ST, Teets, F, Tiemann, JKS, Watkins, A, Woods, H, Yachnin, BJ, Bahl, CD, Bailey-Kellogg, C, Baker, D, Das, R, DiMaio, F, Khare, SD, Kortemme, T, Labonte, JW, Lindorff-Larsen, K, Meiler, J, Schief, W, Schueler-Furman, O, Siegel, JB, Stein, A, Yarov-Yarovoy, V, Kuhlman, B, Leaver-Fay, A, Gront, D, Gray, JJ & Bonneau, R 2021, 'Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks', Nature Communications, vol. 12, 6947. https://doi.org/10.1038/s41467-021-27222-7

APA

Koehler Leman, J., Lyskov, S., Lewis, S. M., Adolf-Bryfogle, J., Alford, R. F., Barlow, K., Ben-Aharon, Z., Farrell, D., Fell, J., Hansen, W. A., Harmalkar, A., Jeliazkov, J., Kuenze, G., Krys, J. D., Ljubetič, A., Loshbaugh, A. L., Maguire, J., Moretti, R., Mulligan, V. K., ... Bonneau, R. (2021). Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks. Nature Communications, 12, [6947]. https://doi.org/10.1038/s41467-021-27222-7

Vancouver

Koehler Leman J, Lyskov S, Lewis SM, Adolf-Bryfogle J, Alford RF, Barlow K et al. Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks. Nature Communications. 2021;12. 6947. https://doi.org/10.1038/s41467-021-27222-7

Author

Koehler Leman, Julia ; Lyskov, Sergey ; Lewis, Steven M. ; Adolf-Bryfogle, Jared ; Alford, Rebecca F. ; Barlow, Kyle ; Ben-Aharon, Ziv ; Farrell, Daniel ; Fell, Jason ; Hansen, William A. ; Harmalkar, Ameya ; Jeliazkov, Jeliazko ; Kuenze, Georg ; Krys, Justyna D. ; Ljubetič, Ajasja ; Loshbaugh, Amanda L. ; Maguire, Jack ; Moretti, Rocco ; Mulligan, Vikram Khipple ; Nance, Morgan L. ; Nguyen, Phuong T. ; Ó Conchúir, Shane ; Roy Burman, Shourya S. ; Samanta, Rituparna ; Smith, Shannon T. ; Teets, Frank ; Tiemann, Johanna K. S. ; Watkins, Andrew ; Woods, Hope ; Yachnin, Brahm J. ; Bahl, Christopher D. ; Bailey-Kellogg, Chris ; Baker, David ; Das, Rhiju ; DiMaio, Frank ; Khare, Sagar D. ; Kortemme, Tanja ; Labonte, Jason W. ; Lindorff-Larsen, Kresten ; Meiler, Jens ; Schief, William ; Schueler-Furman, Ora ; Siegel, Justin B. ; Stein, Amelie ; Yarov-Yarovoy, Vladimir ; Kuhlman, Brian ; Leaver-Fay, Andrew ; Gront, Dominik ; Gray, Jeffrey J. ; Bonneau, Richard. / Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks. In: Nature Communications. 2021 ; Vol. 12.

Bibtex

@article{acd029743c004c3d9bc573b00057b60b,
title = "Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks",
abstract = "Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.",
author = "{Koehler Leman}, Julia and Sergey Lyskov and Lewis, {Steven M.} and Jared Adolf-Bryfogle and Alford, {Rebecca F.} and Kyle Barlow and Ziv Ben-Aharon and Daniel Farrell and Jason Fell and Hansen, {William A.} and Ameya Harmalkar and Jeliazko Jeliazkov and Georg Kuenze and Krys, {Justyna D.} and Ajasja Ljubeti{\v c} and Loshbaugh, {Amanda L.} and Jack Maguire and Rocco Moretti and Mulligan, {Vikram Khipple} and Nance, {Morgan L.} and Nguyen, {Phuong T.} and {{\'O} Conch{\'u}ir}, Shane and {Roy Burman}, {Shourya S.} and Rituparna Samanta and Smith, {Shannon T.} and Frank Teets and Tiemann, {Johanna K. S.} and Andrew Watkins and Hope Woods and Yachnin, {Brahm J.} and Bahl, {Christopher D.} and Chris Bailey-Kellogg and David Baker and Rhiju Das and Frank DiMaio and Khare, {Sagar D.} and Tanja Kortemme and Labonte, {Jason W.} and Kresten Lindorff-Larsen and Jens Meiler and William Schief and Ora Schueler-Furman and Siegel, {Justin B.} and Amelie Stein and Vladimir Yarov-Yarovoy and Brian Kuhlman and Andrew Leaver-Fay and Dominik Gront and Gray, {Jeffrey J.} and Richard Bonneau",
note = "Publisher Copyright: {\textcopyright} 2021, The Author(s).",
year = "2021",
doi = "10.1038/s41467-021-27222-7",
language = "English",
volume = "12",
journal = "Nature Communications",
issn = "2041-1723",
publisher = "nature publishing group",

}

RIS

TY - JOUR

T1 - Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

AU - Koehler Leman, Julia

AU - Lyskov, Sergey

AU - Lewis, Steven M.

AU - Adolf-Bryfogle, Jared

AU - Alford, Rebecca F.

AU - Barlow, Kyle

AU - Ben-Aharon, Ziv

AU - Farrell, Daniel

AU - Fell, Jason

AU - Hansen, William A.

AU - Harmalkar, Ameya

AU - Jeliazkov, Jeliazko

AU - Kuenze, Georg

AU - Krys, Justyna D.

AU - Ljubetič, Ajasja

AU - Loshbaugh, Amanda L.

AU - Maguire, Jack

AU - Moretti, Rocco

AU - Mulligan, Vikram Khipple

AU - Nance, Morgan L.

AU - Nguyen, Phuong T.

AU - Ó Conchúir, Shane

AU - Roy Burman, Shourya S.

AU - Samanta, Rituparna

AU - Smith, Shannon T.

AU - Teets, Frank

AU - Tiemann, Johanna K. S.

AU - Watkins, Andrew

AU - Woods, Hope

AU - Yachnin, Brahm J.

AU - Bahl, Christopher D.

AU - Bailey-Kellogg, Chris

AU - Baker, David

AU - Das, Rhiju

AU - DiMaio, Frank

AU - Khare, Sagar D.

AU - Kortemme, Tanja

AU - Labonte, Jason W.

AU - Lindorff-Larsen, Kresten

AU - Meiler, Jens

AU - Schief, William

AU - Schueler-Furman, Ora

AU - Siegel, Justin B.

AU - Stein, Amelie

AU - Yarov-Yarovoy, Vladimir

AU - Kuhlman, Brian

AU - Leaver-Fay, Andrew

AU - Gront, Dominik

AU - Gray, Jeffrey J.

AU - Bonneau, Richard

N1 - Publisher Copyright: © 2021, The Author(s).

PY - 2021

Y1 - 2021

N2 - Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

AB - Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

U2 - 10.1038/s41467-021-27222-7

DO - 10.1038/s41467-021-27222-7

M3 - Journal article

C2 - 34845212

AN - SCOPUS:85120055073

VL - 12

JO - Nature Communications

JF - Nature Communications

SN - 2041-1723

M1 - 6947

ER -

ID: 288711473