Aging associates with sleep dysfunction as well as brain alterations. However, the association between age-related brain alterations and their subjective sleep changes is less understood. To address this issue, we recorded T1 weighted structural and resting-state functional magnetic resonance imaging from both young (n = 62) and older adults (n = 108). In addition, all participants completed a battery of psychometric tests, including the Pittsburg Sleep Quality Index. We found that the age-related atrophy of cerebral gray matter, hippocampal and thalamic volume were associated with subjective sleep decline, and the atrophy of cerebral gray matter mediated the age effect on sleep. In addition, older adults exhibited decreased functional connectivity within the medial temporal lobe subsystem than their young counterparts. Moreover, there is a significant positive association between sleep and functional connectivity in young but not in older adults. In light of our findings, we suggest a neuropathological model in which age-related brain alterations may partially explain the well-documented declines in sleep with aging.