A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset: Findings From The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge

Publikation: Bidrag til tidsskriftKonferenceabstrakt i tidsskriftForskningfagfællebedømt

  • Arjun D. Desai
  • Francesco Caliva
  • Claudia Iriondo
  • Naji Khosravan
  • Aliasghar Mortazi
  • Sachin Jambawalikar
  • Drew Torigian
  • Jutta Ellerman
  • Mehmet Akcakaya
  • Ulas Bagci
  • Radhika Tibrewala
  • Io Flament
  • Matthew O`Brien
  • Sharmila Majumdar
  • Akshay Pai
  • Sibaji Gaj
  • Mingrui Yang
  • Kunio Nakamura
  • Xiaojuan Li
  • Cem M. Deniz
  • Vladimir Juras
  • Ravinder Regatte
  • Garry E. Gold
  • Brian A. Hargreaves
  • Valentina Pedoia
  • Akshay S. Chaudhari
Purpose: To organize a knee MRI segmentation challenge for characterizing the semantic and clinical efficacy of automatic segmentation methods relevant for monitoring osteoarthritis progression. Methods: A dataset partition consisting of 3D knee MRI from 88 subjects at two timepoints with ground-truth articular (femoral, tibial, patellar) cartilage and meniscus segmentations was standardized. Challenge submissions and a majority-vote ensemble were evaluated using Dice score, average symmetric surface distance, volumetric overlap error, and coefficient of variation on a hold-out test set. Similarities in network segmentations were evaluated using pairwise Dice correlations. Articular cartilage thickness was computed per-scan and longitudinally. Correlation between thickness error and segmentation metrics was measured using Pearson's coefficient. Two empirical upper bounds for ensemble performance were computed using combinations of model outputs that consolidated true positives and true negatives. Results: Six teams (T1-T6) submitted entries for the challenge. No significant differences were observed across all segmentation metrics for all tissues (p=1.0) among the four top-performing networks (T2, T3, T4, T6). Dice correlations between network pairs were high (>0.85). Per-scan thickness errors were negligible among T1-T4 (p=0.99) and longitudinal changes showed minimal bias (
TidsskriftOsteoarthritis and Cartilage Open
Udgave nummerSuppl. 1
Sider (fra-til)5304-5305
StatusUdgivet - 2020
Begivenhed2020 OARSI World Congress on Osteoarthritis - Vienna, Østrig
Varighed: 30 apr. 20203 maj 2020


Konference2020 OARSI World Congress on Osteoarthritis

Bibliografisk note

Submitted to Radiology: Artificial Intelligence


  • eess.IV, cs.CV

ID: 255780231