TY - JOUR

T1 - Design of a deep learning model for automatic scoring of periodic and non-periodic leg movements during sleep validated against multiple human experts

AU - Carvelli, Lorenzo

AU - Olesen, Alexander N.

AU - Brink-Kjær, Andreas

AU - Leary, Eileen B.

AU - Peppard, Paul E.

AU - Mignot, Emmanuel

AU - Sørensen, Helge B.D.

AU - Jennum, Poul

PY - 2020

Y1 - 2020

N2 - Objective: Currently, manual scoring is the gold standard of leg movement scoring (LMs) and periodic LMs (PLMS) in overnight polysomnography (PSG) studies, which is subject to inter-scorer variability. The objective of this study is to design and validate an end-to-end deep learning system for the automatic scoring of LMs and PLMS in sleep. Methods: The deep learning system was developed, validated and tested, with respect to manual annotations by expert technicians on 800 overnight PSGs using a leg electromyography channel. The study includes data from three cohorts, namely, the Wisconsin Sleep Cohort (WSC), Stanford Sleep Cohort (SSC) and MrOS Sleep Study. The performance of the system was further compared against individual expert technicians and existing PLM detectors. Results: The system achieved an F1 score of 0.83, 0.71, and 0.77 for the WSC, SSC, and an ancillary study (Osteoporotic Fractures in Men Study, MrOS) cohorts, respectively. In a total of 60 PSGs from the WSC and the SSC scored by nine expert technicians, the system performed better than two and comparable to seven of the individual scorers with respect to a majority-voting consensus of the remaining scorers. In 60 PSGs from the WSC scored accurately for PLMS, the system outperformed four previous PLM detectors, which were all evaluated on the same data, with an F1 score of 0.85. Conclusions: The proposed system performs better or comparable to individual expert technicians while outperforming previous automatic detectors. Thereby, the study validates fully automatic methods for scoring LMs in sleep.

AB - Objective: Currently, manual scoring is the gold standard of leg movement scoring (LMs) and periodic LMs (PLMS) in overnight polysomnography (PSG) studies, which is subject to inter-scorer variability. The objective of this study is to design and validate an end-to-end deep learning system for the automatic scoring of LMs and PLMS in sleep. Methods: The deep learning system was developed, validated and tested, with respect to manual annotations by expert technicians on 800 overnight PSGs using a leg electromyography channel. The study includes data from three cohorts, namely, the Wisconsin Sleep Cohort (WSC), Stanford Sleep Cohort (SSC) and MrOS Sleep Study. The performance of the system was further compared against individual expert technicians and existing PLM detectors. Results: The system achieved an F1 score of 0.83, 0.71, and 0.77 for the WSC, SSC, and an ancillary study (Osteoporotic Fractures in Men Study, MrOS) cohorts, respectively. In a total of 60 PSGs from the WSC and the SSC scored by nine expert technicians, the system performed better than two and comparable to seven of the individual scorers with respect to a majority-voting consensus of the remaining scorers. In 60 PSGs from the WSC scored accurately for PLMS, the system outperformed four previous PLM detectors, which were all evaluated on the same data, with an F1 score of 0.85. Conclusions: The proposed system performs better or comparable to individual expert technicians while outperforming previous automatic detectors. Thereby, the study validates fully automatic methods for scoring LMs in sleep.

KW - Automatic event detection

KW - Leg movements during sleep

KW - Manual scoring of polysomnography

KW - Periodic leg movements during sleep

KW - Polysomnography

U2 - 10.1016/j.sleep.2019.12.032

DO - 10.1016/j.sleep.2019.12.032

M3 - Journal article

C2 - 32062037

AN - SCOPUS:85079175170

VL - 69

SP - 109

EP - 119

JO - Sleep Medicine

JF - Sleep Medicine

SN - 1389-9457

ER -