Episodic memories are established and maintained by close interplay between hippocampus and other cortical regions, but degradation of a fronto-striatal network has been suggested to be a driving force of memory decline in aging. We wanted to directly address how changes in hippocampal-cortical versus striatal-cortical networks over time impact episodic memory with age. We followed 119 healthy participants (20–83 years) for 3.5 years with repeated tests of episodic verbal memory and magnetic resonance imaging for quantification of functional and structural connectivity and regional brain atrophy. While hippocampal-cortical functional connectivity predicted memory change in young, changes in cortico-striatal functional connectivity were related to change in recall in older adults. Within each age group, effects of functional and structural connectivity were anatomically closely aligned. Interestingly, the relationship between functional connectivity and memory was strongest in the age ranges where the rate of reduction of the relevant brain structure was lowest, implying selective impacts of the different brain events on memory. Together, these findings suggest a partly sequential and partly simultaneous model of brain events underlying cognitive changes in aging, where different functional and structural events are more or less important in various time windows, dismissing a simple uni-factorial view on neurocognitive aging.