The study reports on the associations of infant and childhood anthropometric measurements, early growth, and the combined effect of birth weight and childhood body mass index with older age physical functioning among 1,999 individuals born in 1934–1944 and belonging to the Helsinki Birth Cohort Study. Physical functioning was assessed by the Short Form 36 scale. Anthropometric data from infancy and childhood were retrieved from medical records. The risk of lower Short Form 36 physical functioning at the mean age of 61.6 years was increased for those with birth weight less than 2.5 kg compared with those weighing 3.0–3.5 kg at birth (odds ratio (OR) = 2.73, 95% confidence interval (CI): 1.57, 4.72). The gain in weight from birth to age 2 years was associated with decreased risk of lower physical functioning for a 1-standard deviation increase (OR = 0.84, 95% CI: 0.75, 0.94). The risk of lower physical functioning was highest for individuals with birth weight in the lowest third and body mass index at 11 years of age in the highest third compared with those whose birth weight was in the middle third and body mass index at age 11 years was in the highest third (OR = 3.08, 95% CI: 1.83, 5.19). The increasing prevalence of obesity at all ages and the aging of populations warrant closer investigation of the role of weight trajectories in old age functional decline.