Background: Childhood adversities are known to increase the life-long risk of negative life events and health problems. Less is known about how childhood conditions affect parental empowerment, which is crucial to family well-being and resources. This study aimed to find out how parents’ own childhood adversities predict maternal and paternal empowerment, and how these predictors differ between mothers and fathers. Methods: The study design was cross-sectional. The sample consisted of mothers (n = 571) and fathers (n = 384) of children aged 0–9 years in Finland. Parents were selected using stratified random sampling in 2009. Parental empowerment was measured by the Generic Family Empowerment Scale. Eleven types of childhood adversity were inquired about. Associations were examined by linear regression analysis, and empowerment was predicted by multiple linear regression. Results: Childhood adversities had a negative effect on parental empowerment. The effect was more significant on maternal than paternal empowerment. Adversities were particularly associated with parents’ confidence in managing their children in everyday life. Parental empowerment of both mothers and fathers was affected significantly by their parents’ mental health problems and serious conflicts within their families during childhood. Financial difficulties, parents’ unemployment and bullying at school affected maternal empowerment. Conclusions: Early recognition of adversities and appropriate intervention may help to increase present and future well-being and prevent long-term difficulties of each family member.