Previous research has documented change in white matter tract integrity with increasing age. Both interhemispheric and intrahemispheric tracts that underlie language processing are susceptible to these age-related changes. The aim of the current study was to explore age and white matter integrity in language-related tracts as predictors of cognitive task performance in younger and older adults. To this end, we carried out principal component analyses of white matter tracts and confirmatory factor analysis of neuropsychological measures. We next carried out a series of regression analyses that used white matter components to predict scores on each of the neuropsychological components. For both younger and older adults, age was a significant predictor of processing speed and working memory. However, white matter integrity did not contribute independently toward these models. In older adults only, both age and a white matter component that included the bilateral frontal aslant tract and left superior longitudinal fasciculus were significant predictors of working memory. Taken together, these results extend our understanding of the contributions of language-related white matter structure to cognitive processing and highlight the effects of age-related differences in both frontal and dorsal tracts.