Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods

Research output: Contribution to journal › Journal article › Research › peer-review

Standard

Sleep-spindle detection : crowdsourcing and evaluating performance of experts, non-experts and automated methods. / Warby, Simon C; Wendt, Sabrina L; Welinder, Peter; Munk, Emil G S; Carrillo, Oscar; Sorensen, Helge B D; Jennum, Poul; Peppard, Paul E; Perona, Pietro; Mignot, Emmanuel.

In: Nature Methods, Vol. 11, No. 4, 04.2014, p. 385-392.

Research output: Contribution to journal › Journal article › Research › peer-review

Harvard

Warby, SC, Wendt, SL, Welinder, P, Munk, EGS, Carrillo, O, Sorensen, HBD, Jennum, P, Peppard, PE, Perona, P & Mignot, E 2014, 'Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods', Nature Methods, vol. 11, no. 4, pp. 385-392. https://doi.org/10.1038/nmeth.2855

APA

Warby, S. C., Wendt, S. L., Welinder, P., Munk, E. G. S., Carrillo, O., Sorensen, H. B. D., Jennum, P., Peppard, P. E., Perona, P., & Mignot, E. (2014). Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nature Methods, 11(4), 385-392. https://doi.org/10.1038/nmeth.2855

Vancouver

Warby SC, Wendt SL, Welinder P, Munk EGS, Carrillo O, Sorensen HBD et al. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nature Methods. 2014 Apr;11(4):385-392. https://doi.org/10.1038/nmeth.2855

Author

Warby, Simon C ; Wendt, Sabrina L ; Welinder, Peter ; Munk, Emil G S ; Carrillo, Oscar ; Sorensen, Helge B D ; Jennum, Poul ; Peppard, Paul E ; Perona, Pietro ; Mignot, Emmanuel. / Sleep-spindle detection : crowdsourcing and evaluating performance of experts, non-experts and automated methods. In: Nature Methods. 2014 ; Vol. 11, No. 4. pp. 385-392.

Bibtex

@article{17239aa1377345959ca51d6fd5e112b7,
title = "Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods",
abstract = "Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.",
keywords = "Aged, Algorithms, Automation, Crowdsourcing, Electroencephalography, Humans, Internet, Middle Aged, Sleep Stages",
author = "Warby, {Simon C} and Wendt, {Sabrina L} and Peter Welinder and Munk, {Emil G S} and Oscar Carrillo and Sorensen, {Helge B D} and Poul Jennum and Peppard, {Paul E} and Pietro Perona and Emmanuel Mignot",
year = "2014",
month = apr,
doi = "10.1038/nmeth.2855",
language = "English",
volume = "11",
pages = "385--392",
journal = "Nature Methods",
issn = "1548-7091",
publisher = "nature publishing group",
number = "4",

}

RIS

TY - JOUR

T1 - Sleep-spindle detection

T2 - crowdsourcing and evaluating performance of experts, non-experts and automated methods

AU - Warby, Simon C

AU - Wendt, Sabrina L

AU - Welinder, Peter

AU - Munk, Emil G S

AU - Carrillo, Oscar

AU - Sorensen, Helge B D

AU - Jennum, Poul

AU - Peppard, Paul E

AU - Perona, Pietro

AU - Mignot, Emmanuel

PY - 2014/4

Y1 - 2014/4

N2 - Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

AB - Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

KW - Aged

KW - Algorithms

KW - Automation

KW - Crowdsourcing

KW - Electroencephalography

KW - Humans

KW - Internet

KW - Middle Aged

KW - Sleep Stages

U2 - 10.1038/nmeth.2855

DO - 10.1038/nmeth.2855

M3 - Journal article

C2 - 24562424

VL - 11

SP - 385

EP - 392

JO - Nature Methods

JF - Nature Methods

SN - 1548-7091

IS - 4

ER -

ID: 137620278