It is unclear whether the longitudinal relation between activity participation and cognitive ability is due to preserved differentiation (active individuals have higher initial levels of cognitive ability), or differential preservation (active individuals show less negative change across time). This distinction has never been evaluated after dividing time-varying activity into its two sources of variation: between-person and within-person variability. Further, few studies have investigated how the association between activity participation and cognitive ability may differ from early to older adulthood. Using the PATH Through Life Project, we evaluated whether between- and within-person variation in activity participation was associated with cognitive ability and change within cohorts aged 20–24 years, 40–44 years, and 60–64 years at baseline (n = 7,152) assessed on three occasions over an 8-year interval. Multilevel models indicated that between-person differences in activity significantly predicted baseline cognitive ability for all age cohorts and for each assessed cognitive domain (perceptual speed, short-term memory, working memory, episodic memory, and vocabulary), even after accounting for sex, education, occupational status, and physical and mental health. In each case, greater average participation was associated with higher baseline cognitive ability. However, the size of the relationship involving average activity participation and baseline cognitive ability did not differ across adulthood. Between-person activity and within-person variation in activity level were both not significantly associated with change in cognitive test performance. Results suggest that activity participation is indeed related to cognitive ability across adulthood, but only in relation to the starting value of cognitive ability, and not change over time.