The ability to encode and retrieve spatial and temporal contextual details of episodic memories (context memory) begins to decline at midlife. In the current study, event-related fMRI was used to investigate the neural correlates of context memory decline in healthy middle aged adults (MA) compared with young adults (YA). Participants were scanned while performing easy and hard versions of spatial and temporal context memory tasks. Scans were obtained at encoding and retrieval. Significant reductions in context memory retrieval accuracy were observed in MA, compared with YA. The fMRI results revealed that overall, both groups exhibited similar patterns of brain activity in parahippocampal cortex, ventral occipito-temporal regions and prefrontal cortex (PFC) during encoding. In contrast, at retrieval, there were group differences in ventral occipito-temporal and PFC activity, due to these regions being more activated in MA, compared with YA. Furthermore, only in YA, increased encoding activity in ventrolateral PFC, and increased retrieval activity in occipital cortex, predicted increased retrieval accuracy. In MA, increased retrieval activity in anterior PFC predicted increased retrieval accuracy. These results suggest that there are changes in PFC contributions to context memory at midlife.