The neuroprotective effects of physical activity (PA) are consistently shown in older adults, but the neural substrates, particularly in white matter (WM), are understudied, especially in very old adults with the fastest growth rate and the highest risk of dementia. This study quantified the association between PA and WM integrity in adults over 80. The moderating effects of cardiometabolic conditions, physical functional limitations and WM hyperintensities were also examined, as they can affect PA and brain integrity. Fractional anisotropy (FA) from normal-appearing WM via diffusion tensor imaging and WM hyperintensities were obtained in 90 participants (mean age = 87.4, 51.1% female, 55.6% white) with concurrent objective measures of steps, active energy expenditure (AEE in kcal), duration (min), and intensity (metabolic equivalents, METs) via SenseWear Armband. Clinical adjudication of cognitive status, prevalence of stroke and diabetes, systolic blood pressure, and gait speed were assessed at time of neuroimaging. Participants were on average sedentary (mean ± SD/day: 1766 ± 1345 steps, 202 ± 311 kcal, 211 ± 39 min, 1.8 ± 1.1 METs). Higher steps, AEE and duration, but not intensity, were significantly associated with higher FA. Associations were localized in frontal and temporal areas. Moderating effects of cardiometabolic conditions, physical functional limitations, and WM hyperintensities were not significant. Neither FA nor PA was related to cognitive status. Older adults with a sedentary lifestyle and a wide range of cardiometabolic conditions and physical functional limitations, displayed higher WM integrity in relation to higher PA. Studies of very old adults to quantify the role of PA in reducing dementia burden via WM integrity are warranted.