Executive functions are often considered lynchpin “frontal lobe tasks”, despite accumulating evidence that a broad network of anterior and posterior brain structures supports them. Using a latent variable modelling approach, we assessed whether prefrontal grey matter volumes independently predict executive function performance when statistically differentiated from global atrophy and individual non-frontal lobar volume contributions. We further examined whether fronto-parietal white matter microstructure underlies and independently contributes to executive functions. We developed a latent variable model to decompose lobar grey matter volumes into a global grey matter factor and specific lobar volumes (i.e. prefrontal, parietal, temporal, occipital) that were independent of global grey matter. We then added mean fractional anisotropy (FA) for the superior longitudinal fasciculus (dorsal portion), corpus callosum, and cingulum bundle (dorsal portion) to models that included grey matter volumes related to cognitive variables in previous analyses. Results suggested that the 2-factor model (shifting/inhibition, updating/working memory) plus an information processing speed factor best explained our executive function data in a sample of 202 community dwelling older adults, and was selected as the base measurement model for further analyses. Global grey matter was related to the executive function and speed variables in all four lobar models, but independent contributions of the frontal lobes were not significant. In contrast, when assessing the effect of white matter microstructure, cingulum FA made significant independent contributions to all three executive function and speed variables and corpus callosum FA was independently related to shifting/inhibition and speed. Findings from the current study indicate that while prefrontal grey matter volumes are significantly associated with cognitive neuroscience measures of shifting/inhibition and working memory in healthy older adults, they do not independently predict executive function when statistically isolated from global atrophy and individual non-frontal lobar volume contributions. In contrast, better microstructure of fronto-parietal white matter, namely the corpus callosum and cingulum, continued to predict executive functions after accounting for global grey matter atrophy. These findings contribute to a growing literature suggesting that prefrontal contributions to executive functions cannot be viewed in isolation from more distributed grey and white matter effects in a healthy older adult cohort.