Objective: Elevated cortisol levels with aging have been associated with atrophy of the hippocampus and prefrontal cortex (PFC), as well as with impaired cognitive functions in men. However, coexisting diseases have confounded many studies examining these relationships. Studies in women are lacking. Our objective was to test whether salivary cortisol levels were related to morphology of the hippocampus and the PFC, and to cognitive performance. Design: A cross-sectional study including 200 elderly (55-80 years old) men and women. Method: We used magnetic resonance imaging, tests of episodic-, semantic-, and working memory, visuospatial ability, and cortisol levels in four saliva samples collected during 1 day. Results: Area under the curve (AUC) for cortisol levels was negatively related to cortical surface area of the left anterior cingulate gyrus (caudal P < 0.001; rostral P = 0.006), right lateral orbitofrontal cortex (P = 0.004), and right rostral middle frontal gyrus (P = 0.003). In women, there was also a negative relationship with cortical surface area in the left rostral middle frontal gyrus (P = 0.006). No relationship was found between cortisol levels and hippocampal volume. Conclusion: This study suggests that the structure of the medial PFC is related to cortisol levels in both elderly women and men.