In healthy middle-aged men, endogenous testosterone does not seem to increase risk for cardiovascular disease (CVD). One explanation might be a differential effect of testosterone, and another, interference with oestradiol with respect to specific cardiovascular functions. To investigate these possibilities, we evaluated in a cross-sectional population of 1223 healthy men, aged 46 (6) years, associations between endogenous testosterone, oestradiol and left ventricular structure and function (echocardiography). Testosterone was inversely associated with ejection fraction (EF) and with more sensitive systolic tissue Doppler imaging indices. Oestradiol was positively associated with EF. These associations were confirmed by linear regression analyses, and consistent for calculated free as well as for total sex steroid concentrations. Standardized regression coefficients were −0.13 for testosterone (P < 0.01) and 0.12 for oestradiol (P < 0.01) for the association with EF, in a model which included height, waist circumference, triglycerides, glucose, systolic blood pressure, drug-treated hypertension, heart rate, haematocrit, current smoking, serum sampling time, age and excessive alcohol use. The study suggests an opposite link, albeit modestly, of testosterone and oestradiol with left ventricle systolic function in healthy middle-aged men. The finding provides a partial explanation for the overall neutral effect on CVD of testosterone in healthy middle-aged men.