Objectives Ordinal scales provide means for communicating the severity of a condition, but they are affected by cognitive biases, they introduce statistical problems and they sacrifice resolution. Clinicians discern more details than contained in scales, for example, when assessing radiographs, but clinicians' distinctions are often based on experience-based rules of thumb, that is, heuristics. The objectives of this study are to compare clinicians' heuristic assessments to ordinal grading, to identify case elements that influence clinicians' judgements and to present a method for quantifying heuristic assessments. Design Clinicians were presented with 17 207 random pairs from a set of 1087 knee radiographs. For each pair, the radiograph with more severe osteoarthritis was selected. The Bradley-Terry model was used to calculate an osteoarthritis strength parameter for each radiograph. Similarly, strength parameters were determined for 12 morphological features with five additional features being considered either present or absent. All radiographs were also graded according to conventional ordinal systems (Kellgren-Lawrence and Ahlbäck). Relations between clinicians' judgements and (1) the heuristics-based osteoarthritis strength, (2) conventional ordinal systems and (3) morphological features were investigated. Results Receiver operating characteristic analysis showed that the Bradley-Terry model provided a good description of clinicians' assessments (area under the curve (AUC)=0.97, 95% CI 0.968 to 0.972). Morphological features (AUC=0.90, 95% CI 0.900 to 0.908) provided a superior description of clinicians' choices compared with conventional ordinal systems (AUC=0.88, 95% CI 0.878 to 0.887 and AUC=0.80, 95% CI 0.796 to 0.809) for Ahlbäck and Kellgren-Lawrence, respectively). The features most strongly associated with osteoarthritis strength were medial joint space width, flattening of the medial femoral and tibial condyles, medial osteophytes and alignment. Conclusions Heuristics-based assessments give a better distinction than conventional grading systems of knee osteoarthritis. The example presents a general approach to evaluate which features are part of experts' heuristics. The data suggest that experts discern more details than included in conventional ordinal grading systems. Quantitative heuristic assessments may replace ordinal scales.