Background: The mean total birth rate of the world had been gradually decreasing, with the rate in Japan now at its lowest level internationally. From a public health perspective, it is important to determine the impact of the number of children on all-cause mortality. Methods: A total of 96 311 individuals from the Japan Collaborative Cohort Study were followed from 1988–90 for an average of 14.4 years. Hazard ratios (HRs) with a 95% confidence interval were calculated from proportional hazard models to estimate the risk of all-cause mortality according to the number of children. Results: As of 2006, a total of 18 807 deaths had occurred. Both childless men and women showed higher all-cause mortality risks than those with two children (HR: 1.17 in men and 1.29 in women). Those with one child also showed higher risks (1.13 and 1.16, respectively). Having four or more children among men and five or more children among women also posed a risk (1.16 in men with four children and 1.22 in women with five or more children), showing a U-shaped association between the number of children and all-cause mortality risk. The risk of having only one child seemed evident with the decrease in age among both men and women, while the risk of having many children was apparent with the increase in age. Conclusion: We found a U-shaped association between the number of children and all-cause mortality among both men and women, with the lowest risk among those with two children.