Background: Physical activity plays an important role in prevention of chronic diseases. Animal studies have suggested that lifestyle and exercise habits may have a prenatal origin. Our aim was to assess the role of early growth on leisure time physical activity (LTPA) in later life among 57–70-years-old men and women. Methods: We examined 2003 individuals born in Helsinki, Finland between 1934 and 1944. Of them, 1967 individuals with adequate information on their LTPA in adult life were included in this study. LTPA was assessed by a validated exercise questionnaire (KIHD Study 12 month physical activity history). Subjects’ birth and serial growth measurements were obtained from birth, child welfare and school health records. Results: Participants with higher engagement in LTPA showed a more favourable adult anthropometric and body composition profile than those who were less active. LTPA was positively associated with adult social class. Higher weight and length at birth, and weight at 2 years after adult BMI adjustment, predicted higher intensity of total LTPA (P = 0.04, P = 0.01 and P = 0.03), respectively. Higher height at 2, 7 and 11 years predicted higher intensity of conditioning LTPA (P = 0.01, P = 0.04 and P = 0.004). Higher weight and height at 2, 7 and 11 years predicted higher energy expenditure (EE) of total LTPA (P-values being from 0.01 to 0.03). Furthermore, higher height at 2 and 11 years predicted higher EE of conditioning LTPA (P = 0.02 and P = 0.03). Conclusion: People who as children were taller and weighed more engage more in leisure time physical activity in late adulthood.