Body mass index (BMI) is a highly heritable polygenic trait. It is also affected by various environmental and behavioral risk factors. We used a BMI polygenic risk score (PRS) to study the interplay between the genetic and environmental factors defining BMI. First, we generated a BMI PRS that explained more variance than a BMI genetic risk score (GRS), which was using only genome-wide significant BMI-associated variants (R² = 13.1% compared to 6.1%). Second, we analyzed interactions between BMI PRS and seven environmental factors. We found a significant interaction between physical activity and BMI PRS, even when the well-known effect of the FTO region was excluded from the PRS, using a small dataset of 6,179 samples. Third, we stratified the study population into two risk groups using BMI PRS. The top 22% of the studied populations were included in a high PRS risk group. Engagement in self-reported physical activity was associated with a 1.66 kg/m² decrease in BMI in this group, compared to a 0.84 kg/m² decrease in BMI in the rest of the population. Our results (i) confirm that genetic background strongly affects adult BMI in the general population, (ii) show a non-linear interaction between BMI genetics and physical activity, and (iii) provide a standardized framework for future gene-environment interaction analyses.