Background Body mass index (BMI) shows strong continuity over childhood and adolescence and high childhood BMI is the strongest predictor of adult obesity. Genetic factors strongly contribute to this continuity, but it is still poorly known how their contribution changes over childhood and adolescence. Thus, we used the genetic twin design to estimate the genetic correlations of BMI from infancy to adulthood and compared them to the genetic correlations of height. Methods We pooled individual level data from 25 longitudinal twin cohorts including 38,530 complete twin pairs and having 283,766 longitudinal height and weight measures. The data were analyzed using Cholesky decomposition offering genetic and environmental correlations of BMI and height between all age combinations from 1 to 19 years of age. Results The genetic correlations of BMI and height were stronger than the trait correlations. For BMI, we found that genetic correlations decreased as the age between the assessments increased, a trend that was especially visible from early to middle childhood. In contrast, for height, the genetic correlations were strong between all ages. Age-to-age correlations between environmental factors shared by co-twins were found for BMI in early childhood but disappeared altogether by middle childhood. For height, shared environmental correlations persisted from infancy to adulthood. Conclusions Our results suggest that the genes affecting BMI change over childhood and adolescence leading to decreasing age-to-age genetic correlations. This change is especially visible from early to middle childhood indicating that new genetic factors start to affect BMI in middle childhood. Identifying mediating pathways of these genetic factors can open possibilities for interventions, especially for those children with high genetic predisposition to adult obesity.