Older adult apolipoprotein-E epsilon 4 (APOE-ε4) allele carriers vary considerably in the expression of clinical symptoms of Alzheimer's disease (AD), suggesting that lifestyle or other factors may offer protection from AD-related neurodegeneration. We recently reported that physically active APOE-ε4 allele carriers exhibit a stable cognitive trajectory and protection from hippocampal atrophy over 18 months compared to sedentary ε4 allele carriers. The aim of this study was to examine the interactions between genetic risk for AD and physical activity (PA) on white matter (WM) tract integrity, using diffusion tensor imaging (DTI) MRI, in this cohort of healthy older adults (ages of 65 to 89). Four groups were compared based on the presence or absence of an APOE-ε4 allele (High Risk; Low Risk) and self-reported frequency and intensity of leisure time physical activity (PA) (High PA; Low PA). As predicted, greater levels of PA were associated with greater fractional anisotropy (FA) and lower radial diffusivity in healthy older adults who did not possess the APOE-ε4 allele. However, the effects of PA were reversed in older adults who were at increased genetic risk for AD, resulting in significant interactions between PA and genetic risk in several WM tracts. In the High Risk-Low PA participants, who had exhibited episodic memory decline over the previous 18-months, radial diffusivity was lower and fractional anisotropy was higher, compared to the High Risk-High PA participants. In WM tracts that subserve learning and memory processes, radial diffusivity (DR) was negatively correlated with episodic memory performance in physically inactive APOE-ε4 carriers, whereas DR was positively correlated with episodic memory performance in physically active APOE-ε4 carriers and the two Low Risk groups. The common model of demyelination-induced increase in radial diffusivity cannot directly explain these results. Rather, we hypothesize that PA may protect APOE-ε4 allele carriers from selective neurodegeneration of individual fiber populations at locations of crossing fibers within projection and association WM fiber tracts.