Early life stress (ELS) is associated with increased vulnerability for depression, changes to the corticotropin-releasing hormone (CRH) system and structural and functional changes in hippocampus. Single nucleotide polymorphisms in the CRH receptor 1 (CRHR1) gene interact with ELS to predict depression, cognitive functions and hippocampal activity. Social cognition has been related to hippocampal function and might be crucial for maintaining mental health. However, the interaction of CRHR1 gene variation and ELS on social cognition has not been investigated yet. We assessed social cognition in 502 healthy subjects to test effects of ELS and the CRHR1 gene. Participants were genotyped for rs110402 and rs242924. ELS was assessed by Childhood Trauma Questionnaire, social cognition was measured via Multifaceted Empathy Test and Empathy Quotient. Severity of ELS was associated with decreased emotional, but not cognitive empathy. Subjects with the common homozygous GG GG genotype showed decreased implicit emotional empathy after ELS exposure regardless of its severity. The results reveal that specific CRHR1 polymorphisms moderate the effect of ELS on emotional empathy. Exposure to ELS in combination with a vulnerable genotype results in impaired emotional empathy in adulthood, which might represent an early marker of increased vulnerability after ELS.