Background: We aimed to assess the psychometric properties of the ten-item Anxiety Symptom Scale (ASS) using Rasch analysis. Data from the Lolland-Falster Health Study (LOFUS) were used including ASS data for 16,137 persons aged 18–90 years. Methods: Fit to the Rasch model, ordering of response categories, dimensionality testing, test for differential item functioning, test for local dependency of items, and calculation of reliability were used to evaluate the psychometric characteristics of the ASS. Results: Ordered response categories were achieved for nine of ten items after modifying the original six-point scoring system into a five-point system. After adjustment of the sample size to 500 persons, the ASS fitted the Rasch Model (p = 0.051). Dimensionality testing supported combining the ten items into a total score. No significant differential item functioning was observed for sex, age group, and educational level. Items 1 and 9 and items 4 and 5 demonstrated indication of local dependency. Combining these two item pairs improved the fit of the ASS to the Rasch model. Limitations: The total ASS was poorly targeted for the LOFUS population as the majority of persons were located at the lower end of the construct. This poor alignment could explain the low reliability (PSI 0.49). Analyzing a balanced random sample of responders with high and low ASS scores demonstrated a sufficient PSI (0.81). Conclusions: Despite problems with the measurement structure, the ASS seems to be a promising instrument for measuring anxiety. However, it may prove more reliable for use in clinical samples of respondents.