Numerous cross-sectional studies have used diffusion tensor imaging (DTI) to link age-related differences in white matter (WM) anisotropy and concomitant decrements in cognitive ability. Due to a dearth of longitudinal evidence, the relationship between changes in diffusion properties of WM and cognitive performance remains unclear. Here we examine the relationship between two-year changes in WM organization and cognitive performance in healthy adults (N = 96, age range at baseline = 18–79 years). We used latent change score models (LCSM) to evaluate changes in age-sensitive cognitive abilities — fluid intelligence and associative memory. WM changes were assessed by fractional anisotropy (FA), axial diffusivity (AD), and radial diffusivity (RD) in WM regions that are considered part of established memory networks and exhibited individual differences in change. In modeling change, we postulated reciprocal paths between baseline measures and change factors, within and between WM and cognition domains, and accounted for individual differences in baseline age. Although baseline cross-sectional memory performance was positively associated with FA and negatively with RD, longitudinal effects told an altogether different story. Independent of age, longitudinal improvements in associative memory were significantly associated with linear reductions in FA and increases in RD. The present findings demonstrate the sensitivity of DTI-derived indices to changes in the brain and cognition and affirm the importance of longitudinal models for evaluating brain-cognition relations.