Higher levels of physical fitness or activity (PFA) have been shown to have beneficial effects on cognitive function and grey matter volumes in older adults. However, the relationship between PFA and the brain's white matter (WM) is not yet well established. Here, we aim to provide a comprehensive and systematic review of magnetic resonance imaging studies examining the effects of PFA on the WM of the ageing brain. Twenty-nine studies were included in the review: eleven examined WM volume, fourteen WM lesions, and nine WM microstructure. While many studies found that higher levels of PFA were associated with greater WM volumes, reduced volume or severity of WM lesions, or improved measures of WM microstructure, a number of negative findings have also been published. Meta-analyses of global measures of WM volume and WM lesion volume yielded significant, but small, effect sizes. Overall, we found evidence for cautious support of links between PFA and WM structure, and highlighted key areas for future research including the extent to which the relationship between PFA and WM structure is anatomically specific, the influence of possible confounding factors, and the relationship between PFA, WM and cognition.