Sex differences have been described regarding several aspects of human brain morphology; however, the exact biological mechanisms underlying these differences remain unclear in humans. Women with the complete androgen insensitivity syndrome (CAIS), who lack androgen action in the presence of a 46,XY karyotype, offer the unique opportunity to study isolated effects of sex hormones and sex chromosomes on human neural sexual differentiation. In the present study, we used diffusion tensor imaging to investigate white matter (WM) microstructure in 46,XY women with CAIS (n = 20), 46,XY comparison men (n = 30), and 46,XX comparison women (n = 30). Widespread sex differences in fractional anisotropy (FA), with higher FA in comparison men than in comparison women, were observed. Women with CAIS showed female-typical FA throughout extended WM regions, predominantly due to female-typical radial diffusivity. These findings indicate a predominant role of sex hormones in the sexual differentiation of WM microstructure, although sex chromosome genes and/or masculinizing androgen effects not mediated by the androgen receptor might also play a role.