We investigated the association between cadmium in blood and the concentration of the prostate specific antigen (PSA) in semen, including the modifying effects of zinc or the CAG polymorphism in the androgen receptor (AR). Blood and semen samples were collected from 504 partners of pregnant women in Greenland, Poland and Ukraine. We found an inverse trend between cadmium and PSA (log(β) = -0.121, 95% confidence interval (CI): -0.213; -0.029, P = 0.0103) in Greenlandic men. Similar results were observed in men with a high number of CAG repeats (CAG 24) (log(β) = -0.231, 95% CI: -0.363; -0.098, P = 0.0009). Inverse trends between cadmium and PSA were found when semen zinc concentrations were below the median value for men from Ukraine and Greenland. These outcomes suggest that cadmium may impair prostate function, as measured by PSA in semen, while high zinc levels and a low number of CAG repeats protects against this action.