It is debated whether sex differences in adiponectin and leptin are due to sex differences in body fat distribution. In this cross-sectional analysis of the Netherlands Epidemiology of Obesity study, associations of measures of body fat and sex with serum adiponectin and leptin concentrations were examined using linear regression analysis (n = 6494, VAT: n = 2516). Sex differences were additionally adjusted for the measure of body fat that was most strongly associated with adiponectin or leptin concentrations. Median adiponectin concentrations in women and men were 10.5 mg/L (IQR, interquartile range: 7.7–13.9) and 6.1 mg/L (IQR: 4.5–8.2), mean difference 4.6 mg/L (95% CI: 4.3, 4.9). Median leptin concentrations in women and men were 19.2 μg/L (IQR: 11.5–30.0) and 7.1 μg/L (IQR: 4.6–11.1), mean difference 15.1 μg/L (95% CI: 14.4, 15.8). VAT was most strongly associated with adiponectin, total body fat percentage was most strongly associated with leptin. After adjustment for VAT, women had 3.8 mg/L (95% CI: 3.3, 4.3) higher adiponectin than men. After adjustment for total body fat percentage, leptin concentrations in women were 0.4 μg/L lower than in men (95% CI: −1.2, 2.0). One genetic variant (rs4731420) was associated with extreme leptin concentrations (>100 μg/L) in women: odds ratio 2.8 (95% CI: 1.7, 4.6). Total body fat percentage was strongly associated with leptin concentrations. Higher leptin concentrations in women than in men were completely explained by differences in total body fat percentage. Visceral fat was associated with adiponectin concentrations, and did not completely explain higher adiponectin concentrations in women than in men.