Aging is associated with reduced resources needed to perform difficult cognitive tasks, but the neural underpinnings are not well understood, especially as there is scant evidence linking functional brain differences to aging cognition. Therefore, the current study examined modulation of fMRI activation from easier to harder spatial distance judgments across a large lifespan sample (N=161; ages 20–94) to identify when in the lifespan modulation to difficulty begins to show deficits and if age-related modulation predicts cognition. Analyses revealed two sets of regions in which modulation increased with difficulty due to either more activation (positive modulation) or more deactivation (negative modulation) to difficulty. These two networks evidenced differential aging trajectories: a right-lateralized fronto-parietal network that decreased in modulation to difficulty between middle- and older-age, and a network of regions in ventromedial prefrontal cortex, posterior cingulate, left angular and middle frontal gyri that showed decreased modulation at the transition from younger to middle-age. Critically, older adults who maintained negative modulation to difficulty showed higher task accuracy. Further, individuals who showed greater coupling between positive and negative modulation performed better on a fluid reasoning task. Age-related preservation of coupled modulation in both cognitive control regions and regions typically associated with default network may be a salient marker of how the brain adapts to maintain cognitive function as we age.