Sleep has been hypothesised to facilitate waste clearance from the brain. We aimed to determine whether sleep is associated with perivascular spaces on brain magnetic resonance imaging (MRI), a potential marker of impaired brain waste clearance, in a population-based cohort of middle-aged and elderly people. In 559 participants (mean [SD] age 62 [6] years, 52% women) from the population-based Rotterdam Study, we measured total sleep time, sleep onset latency, wake after sleep onset and sleep efficiency with actigraphy and polysomnography. Perivascular space load was determined with brain MRI in four regions (centrum semiovale, basal ganglia, hippocampus, and midbrain) via a validated machine learning algorithm using T2-weighted MR images. Associations between sleep characteristics and perivascular space load were analysed with zero-inflated negative binomial regression models adjusted for various confounders. We found that higher actigraphy-estimated sleep efficiency was associated with a higher perivascular space load in the centrum semiovale (odds ratio 1.10, 95% confidence interval 1.04–1.16, p = 0.0008). No other actigraphic or polysomnographic sleep characteristics were associated with perivascular space load in other brain regions. We conclude that, contrary to our hypothesis, associations of sleep with perivascular space load in this middle-aged and elderly population remained limited to an association of a high actigraphy-estimated sleep efficiency with a higher perivascular space load in the centrum semiovale.