Principles

Realistic datasets:

  • including multiple strains per species
  • including microdiversity
  • isolate sequences not represented in public databases
  • realistic sequencing characteristics, complexities and abundance distributions

Reproducible predictions:

  • codes will be made public after the contest
  • only tools with reproducible results will be considered to be included in a publication
  • other tools can be tested on performance for developers

Community-driven effort:

  • incorporate as much feedback as possible
  • metrics for evaluations will be discussed and selected in a meeting after the contest by a larger group of experts

Continuous value:

  • continuously operating framework
  • allows future automated benchmarking procedures, facilitating tool development
  • periodically replaced test data