Patients with very low sperm count through direct sperm examination can exhibit extreme oligozoospermia or cryptozoospermia (after centrifugation). The management of these patients is a real challenge for both clinicians and biologists. In this retrospective and comparative cohort study, we compared the andrological phenotype of patients with extreme alterations of spermatogenesis and assessed whether the origin of spermatozoa (testicular or ejaculate) had any influence on intracytoplasmic sperm injection (ICSI) outcomes. A total of 161 ICSI cycles were performed using ejaculated spermatozoa from 75 patients with extreme oligozoospermia (EOS) or cryptozoospermia (CS) and 150 ICSI cycles using extracted testicular spermatozoa from 74 patients with nonobstructive azoospermia (NOA). Physical, hormonal, ultrasound assessments, and ICSI outcomes were performed in each group. Cryptorchidism was significantly more frequent in the NOA group (60.8% vs. 22.6%, p = 0.001). FSH levels were significantly higher [18.9 IU/L (5.9-27.0) vs. 15.3 IU/L (9.0-46.5), p = 0.001] and the majority of inhibin B levels measured were found mostly undetectable in the NOA group as compared to EOS/CS group (31.1% vs. 10.7%, p = 0.0004). Moreover, we found no significant differences in the respect to the fertilization rates (48.9% and 43.3%, p = 0.43), implantation rates (17.4% and 15.9%, p = 0.77), and percentage of top quality embryo (22.4% and 20.4%, p = 0.73) between the two groups. The clinical pregnancy rates per embryo transferred were comparable in both groups (28.3% and 27.4%, p = 0.89). In this study, we showed for the first time a different andrological phenotype between EOS/CS and NOA groups. Indeed, cryptorchidism was significantly more frequent with more severe endocrine parameters found in the NOA group. These results reflect a more profound alteration in spermatogenesis in NOA patients. However, there was no difference in ICSI outcomes between NOA and EOS/CS groups.