TY - JOUR

T1 - A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results

AU - Sandau, Nicolai

AU - Aagaard, Thomas Vedste

AU - Hróbjartsson, Asbjørn

AU - Harris, Ian A.

AU - Brorson, Stig

N1 - Publisher Copyright: © 2023 Elsevier Inc.

PY - 2023

Y1 - 2023

N2 - Objectives: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. Study Design and Setting: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. Results: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%–78%, quality of life; 10%–24%; adverse events 10%–13%). Conclusion: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

AB - Objectives: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. Study Design and Setting: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. Results: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%–78%, quality of life; 10%–24%; adverse events 10%–13%). Conclusion: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

KW - Meta-analysis

KW - Methodological flexibility

KW - Protocol

KW - Proximal humerus fractures

KW - Variation of results

KW - Vibration of effects

U2 - 10.1016/j.jclinepi.2023.02.002

DO - 10.1016/j.jclinepi.2023.02.002

M3 - Journal article

C2 - 36822442

AN - SCOPUS:85151933379

VL - 156

SP - 95

EP - 104

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -