Although cerebrovascular factors are the second leading cause of cognitive impairment and dementia in elderly, the precise spatial and temporal trajectories of vascular decline in aging have not been fully characterized. With an advanced cerebrovascular reactivity (CVR) MRI technique that specifically informs vascular stiffness and dilatory ability of cerebral vessels, we present four-year longitudinal CVR data measured in 116 healthy individuals (20–88 years of age). Our data revealed a spatial heterogeneity in vascular decline in aging (p=0.003), in that temporal lobe showed the fastest rate of longitudinal CVR decline, followed by parietal and frontal lobes. The rate of CVR decline was also age-dependent. Middle age, not older age, manifested the fastest rate of longitudinal CVR decline (p<0.05). Longitudinal changes in CVR were associated with changes in processing speed (p=0.031) and episodic memory (p=0.022), but not with working memory or reasoning. The rate of longitudinal CVR change was not different between hypertensive and normotensive participants. However, cross-sectionally, individuals with hypertension revealed in a lower CVR compared to normotensive participants (p=0.016). These findings help elucidate age-related decline in brain hemodynamics and support CVR as a non-invasive biomarker in evaluating cerebrovascular conditions in elderly individuals.