Research on relationships between personality and cognitive abilities has so far resulted in inconsistent findings regarding the strength of the associations. Moreover, relationships have rarely been compared longitudinally and bidirectionally between midlife versus late-life cohorts by considering different personality traits as well as multiple cognitive domains over a long-term follow-up period. We hypothesize that the interplay between the “Big Five” personality traits (Neuroticism, Extraversion, Openness to Experience, Agreeableness and Conscientiousness) and cognitive abilities (information processing speed, crystallized intelligence, fluid intelligence) may change from midlife to old age due to age-associated changes in cognitive and personality plasticity. We used data from the German Interdisciplinary Longitudinal Study of Adult Development (ILSE study; n = 1,002). Participants were either born in 1950/52 (midlife sample, n = 502) or in 1930/32 (late-life sample, n = 500) and followed up for up to 12 years. Based on bivariate latent change score regression models (adjusted for gender, education, self-rated and physician-rated health), we observed that, apart from very few exceptions, the intervariable cross-lagged associations between personality traits and cognitive abilities were generally similar between cohorts. Moreover, in case of neuroticism, extraversion, and openness, the effects of cognitive abilities on change in personality were stronger than the reversed effects. Our findings thus suggest that the so far predominant perspective of personality in middle adulthood and late-life as a predictor, rather than as an outcome, of cognitive abilities needs more differentiation and reconsideration.