Background Infections and activated immune responses can affect the brain through several pathways that might also affect cognition. However, no large-scale study has previously investigated the effect of infections on the general cognitive ability in the general population. Methods Danish nationwide registers were linked to establish a cohort of all 161,696 male conscripts during the years 2006–2012 who were tested for cognitive ability, which was based on logical, verbal, numerical and spatial reasoning at a mean age of 19.4 years. Test scores were converted to a mean of 100.00 and with a standard deviation (SD) of 15. Data were analyzed as a cohort study with severe infections requiring hospitalization as exposure using linear regression. Results Adjusted effect sizes were calculated with non-exposure to severe infections as reference, ranging from 0.12 SD to 0.63 SD on general cognitive ability. A prior infection was associated with significantly lower cognitive ability by a mean of 1.76 (95%CI: -1.92 to -1.61; corresponding to 0.12 SD). The cognitive ability was affected the most by the temporal proximity of the last infection (P<0.001) and by the severity of infection measured by days of admission (P<0.001). The number of infections were associated with decreased cognitive ability in a dose-response relationship, and highest mean differences were found for ≥10 hospital contacts for infections (Mean: -5.54; 95%CI: -7.20 to -3.89; corresponding to 0.37 SD), and for ≥5 different types of infections (Mean: -9.44; 95%CI: -13.2 to -5.69; corresponding to 0.63 SD). Hospital contacts with infections had occurred in 35% of the individuals prior to conscription. Conclusions Independent of a wide range of possible confounders, significant associations between infections and cognitive ability were observed. Infections or related immune responses might directly affect the cognitive ability; however, associated heritable and environmental factors might also account for the lowered cognitive ability.