Background: Predictors of attention problems remain uncertain. Here we distinguish prematurity from small (birth weight) for gestational age (SGA). Method: A total of 1437 children were studied between 0 and 6 years. Gender differences and indirect perinatal effects (via 20-month head circumference and cognition) were considered for age 6 attention problems. Results: Boys, preterms, and SGA children were all at increased risk for attention problems. Indirect perinatal effects differed between boys and girls. Conclusions: The routes leading to attention problems seem to differ for SGA and preterm children. SGA appears to reduce brain volume while prematurity alters brain function. Although less frequent, female attention problems are more strongly predicted by prematurity and cognitive dysfunction.