Maternal exposure to ambient air pollution has increasingly been linked to congenital heart defects (CHDs). The objective of this study was to evaluate whether high levels of maternal exposure to PM2.5 and PM10 are related to increased risk of CHDs in Wuhan, China. We conducted a cohort study with a total of 105,988 live-born infants, stillbirths, and fetal deaths. The study included mothers living in the urban district of Wuhan during pregnancy over the 2-year period from 10 June 2011 to 9 June 2013. For each study participant, we assigned 1-month and 1-week averages of PM10 and PM2.5 exposure based on measurements obtained from the nearest exposure monitor to the living residence of mothers during their early pregnancy period. Logistic regression analyses were conducted to calculate the adjusted odds ratios (aORs) and 95% confidence intervals (CI) for the association between exposure to these ambient air pollutants during early pregnancy and CHDs. We observed an increased risk of CHDs, particularly ventricular septal defect (VSD), with increasing PM2.5 exposure. Using 1-week averages, we also observed significant monotonically increasing associations between PM2.5 exposure during weeks 7–10 of pregnancy and risk of VSD, with aORs ranging from 1.11 to 1.17 (95% CI: 1.02–1.20, 1.03–1.22, 1.05–1.24, and 1.08–1.26 separately) per a 10 μg/m3 change in PM2.5 concentration. Our study contributes to the small body of knowledge regarding the association between in utero exposure to air pollution and CHDs, but confirmation of these associations will be needed in future studies.