Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta. / Engberg, Morten; Lönn, Lars; Konge, Lars; Mikkelsen, Søren; Hörer, Tal; Lindgren, Hans; Søvik, Edmund; Svendsen, Morten Bo; Frendø, Martin; Taudorf, Mikkel; Russell, Lene.
In: Journal of Trauma and Acute Care Surgery, Vol. 91, No. 4, 2021, p. 663-671.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta
AU - Engberg, Morten
AU - Lönn, Lars
AU - Konge, Lars
AU - Mikkelsen, Søren
AU - Hörer, Tal
AU - Lindgren, Hans
AU - Søvik, Edmund
AU - Svendsen, Morten Bo
AU - Frendø, Martin
AU - Taudorf, Mikkel
AU - Russell, Lene
N1 - Publisher Copyright: © 2021 Wolters Kluwer Health, Inc. All rights reserved.
PY - 2021
Y1 - 2021
N2 - BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. RESULTS Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. CONCLUSION Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.
AB - BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. RESULTS Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. CONCLUSION Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.
KW - clinical competence
KW - Resuscitative endovascular balloon occlusion of the aorta
KW - simulation
KW - skills
KW - validation
KW - validity evidence
KW - vascular access, assessment tool
U2 - 10.1097/TA.0000000000003338
DO - 10.1097/TA.0000000000003338
M3 - Journal article
C2 - 34225347
AN - SCOPUS:85116808410
VL - 91
SP - 663
EP - 671
JO - Journal of Trauma
JF - Journal of Trauma
SN - 2163-0755
IS - 4
ER -
ID: 284198679