Physical frailty increases the risk of future activity limitation, which in turn, compromises independent living of older people and limits their healthspan. Thus, we seek to identify moderators and mediators of the effect of physical frailty on activity limitation change in older people, including gender- and age-specific effects. In a longitudinal study using data from waves 2, 4, and 6 of the English Longitudinal Study of Ageing, unique physical frailty factor scores of 4638 respondents aged 65 to 89 years are obtained from confirmatory factor analysis of physical frailty, which is specified by three indicators, namely slowness, weakness, and exhaustion. Using a series of autoregressive cross-lagged models, we estimate the effect of physical frailty factor score on activity limitation change, including its moderation by social conditions, and indirect effects through physical and psychological conditions. We find that the effect of physical frailty on activity limitation change is significantly stronger with older age, while it has significant indirect effects through low physical activity, depressive symptoms, and cognitive impairment. In turn, indirect effects of physical frailty through low physical activity and cognitive impairment are stronger with older age. Sensitivity analyses suggest that these effects vary in their robustness to unmeasured confounding. We conclude that low physical activity, depressive symptoms, and cognitive impairment are potentially modifiable mediators on pathways from physical frailty to activity limitation in older people, including those who are very old. This evidence offers support for population-level interventions that target these conditions, to mitigate the effect of physical frailty on activity limitation, and thereby enhance healthspan.