Very few studies have assessed whether socioeconomic and psychosocial adversity during childhood are associated with objective measures of aging later in life. We assessed associations of socioeconomic position (SEP) and total psychosocial adversity during childhood, with objectively measured cognitive and physical capability in women during midlife. Adverse childhood experiences were retrospectively reported at mean ages 28–30 years in women from the Avon Longitudinal Study of Parents And Children (N = 2,221). We investigated associations of childhood SEP and total psychosocial adversity, with composite measures of cognitive and physical capability at mean age 51 years. There was evidence that, compared with participants whose fathers had professional occupations, participants whose fathers had managerial/technical, skilled nonmanual, skilled manual, and partly or unskilled manual occupations had, on average, lower physical and cognitive capability. There was a clear trend for increasing magnitudes of association with lowering childhood SEP. There was also evidence that greater total psychosocial adversity in childhood was associated with lower physical capability. Total psychosocial adversity in childhood was not associated with cognitive capability. Lower SEP in childhood is detrimental to cognitive and physical capability in midlife, at least in part, independently of subsequent SEP in adulthood. Greater psychosocial adversity in childhood is associated with poorer physical capability, independently of social disadvantage in childhood. Our findings highlight the need for interventions to both identify and support children experiencing socioeconomic or psychosocial of adversity as early as possible.