Slower processing speed (PS), a highly robust feature of cognitive aging, is associated with white matter (WM) deterioration and gray matter volume (GMV) loss. Traditional linear regression models assume a constant relationship between brain structure and cognition over time. To probe for variation in the association between WM and GMV and PS over time, we used a novel sparse varying coefficient model on data collected from 126 relatively healthy older adults (67 females, aged 58–85 years) evaluated with MRI and a standardized neuropsychological test-battery. We found that WM microstructural differences indexed by fractional anisotropy values in the fronto-striatal tracts (internal and external capsule) showed a stronger association with PS before the age of 70 years. Contrastingly, GMV values of the left putamen and middle occipital gyrus were more strongly correlated with PS after 70 years. Additionally, within GM and WM compartments, there was heterogeneity in the temporal sequence in which different cortical and subcortical elements were most strongly associated with PS. Together, these observations provide a more nuanced account of the relationships between different structural components of the aging brain and processing speed, a key cognitive domain affected in relatively healthy older adults.