TY - UNPB

T1 - Biomedical image analysis competitions

T2 - The state of current participation practice

AU - Eisenmann, Matthias

AU - Reinke, Annika

AU - Weru, Vivienn

AU - Tizabi, Minu Dietlinde

AU - Isensee, Fabian

AU - Adler, Tim J.

AU - Godau, Patrick

AU - Cheplygina, Veronika

AU - Kozubek, Michal

AU - Ali, Sharib

AU - Gupta, Anubha

AU - Kybic, Jan

AU - Noble, Alison

AU - Solórzano, Carlos Ortiz de

AU - Pachade, Samiksha

AU - Petitjean, Caroline

AU - Sage, Daniel

AU - Wei, Donglai

AU - Wilden, Elizabeth

AU - Alapatt, Deepak

AU - Andrearczyk, Vincent

AU - Baid, Ujjwal

AU - Bakas, Spyridon

AU - Balu, Niranjan

AU - Bano, Sophia

AU - Bawa, Vivek Singh

AU - Bernal, Jorge

AU - Bodenstedt, Sebastian

AU - Casella, Alessandro

AU - Choi, Jinwook

AU - Commowick, Olivier

AU - Daum, Marie

AU - Depeursinge, Adrien

AU - Dorent, Reuben

AU - Egger, Jan

AU - Eichhorn, Hannah

AU - Engelhardt, Sandy

AU - Ganz, Melanie

AU - Girard, Gabriel

AU - Hansen, Lasse

AU - Heinrich, Mattias

AU - Heller, Nicholas

AU - Hering, Alessa

AU - Huaulmé, Arnaud

AU - Kim, Hyunjeong

AU - Thambawita, Vajira

AU - Zhao, Xin

AU - Lund, Christina B.

AU - Ren, Jintao

AU - Yang, Lin

AU - MICCAI challenge collaboration

PY - 2022

Y1 - 2022

N2 - The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

AB - The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

KW - cs.CV

KW - cs.LG

M3 - Preprint

SP - 1_30

BT - Biomedical image analysis competitions

PB - arXiv.org

ER -