BACKGROUND: Monitoring disease activity over time is a prerequisite for clinical practice and research. Valid and reliable outcome measurements instruments (OMIs) and staging systems provide researchers and clinicians with benchmark tools to assess the primary and secondary outcomes of interventional trials and to guide treatment selection properly.

OBJECTIVES: To investigate interrater reliability and agreement in instruments currently used in Hidradenitis Suppurativa (HS) with HS-experienced dermatologists being the rater population of interest.

METHODS: In a prospective completely balanced design, 24 HS patients underwent a physical examination by 12 raters (288 assessments) using nine instruments; analysed using generalised linear mixed models.

RESULTS: For the staging systems, the study found good interrater reliability for Hurley staging in the axillae and gluteal region, moderate interrater reliability for Hurley staging in the groin and Physician's Global Assessment, and fair interrater reliability for Hurley staging refined and International HS Severity Scoring System. For all the tested OMIs, the observed intervals for limits of agreement were very wide relative to the ranges of the scales.

CONCLUSION: The very wide intervals for limits of agreement imply that substantial changes are needed in clinical research in order to rule out measurement error. The results illustrate a difficulty, even for experienced HS-experts, to agree on the type and number of lesions when evaluating disease severity. The apparent caveats call for global efforts, such as the HIdradenitis SuppuraTiva cORe outcomes set International Collaboration (HISTORIC) to reach consensus on how valid HS physical signs are best measured reliably in randomised trials. This article is protected by copyright. All rights reserved.