Age-related deficits in context memory may arise from neural changes underlying both encoding and retrieval of context information. Although age-related functional changes in the brain regions supporting context memory begin at midlife, little is known about the functional changes with age that support context memory encoding and retrieval across the adult lifespan. We investigated how age-related functional changes support context memory across the adult lifespan by assessing linear changes with age during successful context encoding and retrieval. Using functional magnetic resonance imaging (fMRI), we compared young, middle-aged and older adults during both encoding and retrieval of spatial and temporal details of faces. Multivariate behavioral partial least squares (B-PLS) analysis of fMRI data identified a pattern of whole-brain activity that correlated with a linear age term and a pattern of whole-brain activity that was associated with an age-by-memory phase (encoding vs. retrieval) interaction. Further investigation of this latter effect identified three main findings: 1) reduced phase-related modulation in bilateral fusiform gyrus, left superior/anterior frontal gyrus and right inferior frontal gyrus that started at midlife and continued to older age, 2) reduced phase-related modulation in bilateral inferior parietal lobule that occurred only in older age, and 3) changes in phase-related modulation in older but not younger adults in left middle frontal gyrus and bilateral parahippocampal gyrus that was indicative of age-related over-recruitment. We conclude that age-related reductions in context memory arise in midlife and are related to changes in perceptual recollection and changes in fronto-parietal retrieval monitoring.