The number of cancer survivors continues to increase worldwide. Many of these survivors have had children of their own. It is less well-known whether radiation therapy or chemotherapy could affect the risk of stillbirth and neonatal death for these children. To explore this research questions, we identified all women diagnosed with cancer between 1958 and 2012 from the Swedish Cancer Register and they were further linked to the Swedish Medical Birth Register to identify their subsequent child birth between 1973 and 2012. Multivariate logistic regression was used to estimate odds ratios and 95% confidence intervals for the association between stillbirth and neonatal death and maternal cancer diagnosis. As compared to the children without maternal cancer, the risk of stillbirth was significantly higher among children of female cancer survivors born within three years after cancer diagnosis with an OR of 1.92 (95% CI 1.03–3.57). The incidence of neonatal death did not show a significant change. For women with more than one pregnancy after cancer diagnosis, the risk of stillbirth and neonatal death was lower for the second child birth compared to the first child birth. Our study suggested that the risk of stillbirth was negatively associated with the time after cancer diagnosis, providing evidence that the adverse effect associated with cancer treatment may diminish with time.