Previous studies that focused on age-related changes in the performance of theory of mind (ToM) in older adults have provided conflicting results. ToM consists of cognitive and affective mentalizing processes. Mixed calculation of different aspects of ToM may have contributed to the conflicting results and fails to detect the specific age effects on ToM. The current study investigated the age-related changes on stories tasks that specifically assess cognitive versus affective ToM in 42 young-old adults and 32 old-old adults, compared to 32 young adults. A factor analysis revealed that the ToM stories tasks could be classified into three components. Both of the two older adult groups performed worse than young adult group on cognitive ToM stories tasks (p < .001). However, older adults performed nearly the same as young adults on affective ToM stories tasks (p > .05). Moreover, performance on executive inhibition, measured by the Hayling test, was only correlated with cognitive ToM tasks (β = −0.318, R2 = 0.101), but not with affective ToM tasks. The results reveal a greater age effect on cognitive compared to affective ToM. Rather than a general decline of ToM, older adults show selective compromised performance on cognitive ToM tasks, while relatively intact performance on affective ToM stories tasks. The dissociable correlation between cognitive versus affective ToM with inhibitive control further confirms the dissociation of cognitive and affective ToM.