The aim of this cross-sectional study was to assess the factors associated with the impact of oral health on the quality of life in a sample of 504 Brazilian independent elderly. Data collection included oral examinations and structured interviews. The simplified form of the Oral Health Impact Profile (OHIP-14) was used to measure OHRQoL. Information on sociodemographic characteristics, use of dental services, and subjective measures of health was collected. Poisson regression within a hierarchical model was used to data analyses. The following variables were associated with a negative impact on OHRQoL: female gender (PR = 1.40; CI 95%: 1.11-1.77); lower class (PR = 1.58; CI 95%: 1.13-2.20); up to 3 occluding pairs of posterior teeth (PR = 1.88; CI 95%: 1.13-3.14); at least one untreated caries (PR = 1.28; CI 95%: 1.06-1.54); curative reasons for the last dental appointment (PR = 1.52; CI 95%: 1.15-2.00); poor self-perception of oral health (PR = 2.49; CI 95%: 1.92-3.24); and poor perception of dental care provided (PR = 1.34; CI 95%: 1.12-1.59). The younger elderly also noticed this negative impact. These findings showed that the clinical, sociodemographic, and subjective factors evaluated exerted a negative impact on OHRQoL in elderly people. Health authorities must address all these factors when planning interventions on oral health for this population.