The relationship between epididymis ultrasonography (US) and infertility is poorly defined probably owing to lack of objective and reproducible criteria of US evaluation. Here, we evaluated US size of testes, caput and of corpus epididymis in infertile men: 165 with total sperm count ≥39 × 106, 187 with total sperm count <39 × 106 and 75 azoospermic men. Blood levels of follicle stimulating hormone (FSH) and of total testosterone were also evaluated. US measures obtained using a high-frequency (12 MHz) linear array transducer, included the mean value of bilateral testicular volumes (mL) (Testes-M), of bilateral longitudinal diameter of caput epididymis (mm) (Caput-M) and of the bilateral antero-posterior diameter of the corpus measured on a longitudinal scan (mm) (Corpus-M). Testicular histology of azoospermic men was obtained and the percentage of seminiferous tubules with elongated spermatids (%T) was used to classify cases with normal spermatogenesis (obstructive azoospermia) (n = 17; %T ≥ 80), or with deranged spermatogenesis (n = 58; %T ≤ 33). Caput-M was correlated with Testes-M (p = 0.0003; r = 0.17) and with FSH serum levels (p = 0.024; r = −0.14) but not with semen parameters. Caput-M but not Corpus-M values resulted greater in obstructive azoospermia compared with other groups, but difference was not significant. Cut-off values of Testes-M, Caput-M and of FSH correctly classified cases of obstructive azoospermia (AUC > 0.5). A patient with FSH < 7.8 IU/mL had a 63.6% chance (CI 40.1–83.2%) of being affected by obstructive azoospermia. US Caput-M ≥10.85 mm, which represented the cut-off value with the highest combination of sensitivity (58.8%, CI 32.9–81.6%) and specificity (91.4%, CI 81.0–97.1%) applied in cases with FSH < 7.8 IU/mL increased the probability for obstructive azoospermia from 63.6% up to 92.3% (CI 76.5–98.8%). US evaluation of the caput epididymis diameter helped in predicting the obstructive origin of azoospermia when FSH was not increased, whereas it was not relevant in non-azoospermic men.