Greater physical activity and cardiorespiratory fitness are associated with reduced age-related cognitive decline and lower risk for dementia. However, significant gaps remain in the understanding of how physical activity and fitness protect the brain from adverse effects of brain aging. The primary goal of the current study was to empirically evaluate the independent relationships between physical activity and fitness with functional brain health among healthy older adults, as measured by the functional connectivity of cognitively and clinically relevant resting state networks. To build context for fitness and physical activity associations in older adults, we first demonstrate that young adults have greater within-network functional connectivity across a broad range of cortical association networks. Based on these results and previous research, we predicted that individual differences in fitness and physical activity would be most strongly associated with functional integrity of the networks most sensitive to aging. Consistent with this prediction, and extending on previous research, we showed that cardiorespiratory fitness has a positive relationship with functional connectivity of several cortical networks associated with age-related decline, and effects were strongest in the default mode network (DMN). Furthermore, our results suggest that the positive association of fitness with brain function can occur independent of habitual physical activity. Overall, our findings provide further support that cardiorespiratory fitness is an important factor in moderating the adverse effects of aging on cognitively and clinically relevant functional brain networks.