Assessing emotional states of dairy calves is an essential part of welfare assessment, but standardized protocols are absent. The present study aims at assessing the emotional states of dairy calves and establishing a reliable standard procedure with Qualitative Behavioral Assessment (QBA) and 20 defined terms. Video material was used to compare multiple observer results. Further, live observations were performed on 49 dairy herds in Denmark and Italy. Principal Component Analysis (PCA) identified observer agreement and QBA dimensions (PC). For achieving overall welfare judgment, PC1-scores were turned into theWelfare Quality (WQ) criterion ‘Positive Emotional State’. Finally, farm factors’ influence on the WQ criterion was evaluated by mixed linear models. PCA summarized QBA descriptors as PC1 ‘Valence’ and PC2 ‘Arousal’ (explained variation 40.3% and 13.3%). The highest positive descriptor loadings on PC1 was Happy (0.92) and Nervous (0.72) on PC2. The WQ-criterion score (WQ-C12) was on average 51.1 ± 9.0 points (0: Worst to 100: Excellent state) and ‘Number of calves’, ‘Farming style’, and ‘Breed’ explained 18% of the variability of it. We conclude that the 20 terms achieved a high portion of explained variation providing a differentiated view on the emotional state of calves. The defined term list proved to need good training for observer agreement.