Visual neglect is a heterogeneous, multi-component syndrome resulting from right hemisphere damage. Neglect patients do not pay attention to events occurring on their left side, and have a poor functional outcome. The intra-hemispheric location of lesions producing neglect is debated, because studies using different methods reported different locations in the grey matter and in the white matter of the right hemisphere. These reported locations show various patterns of overlapping with the fronto-parietal attention networks demonstrated by functional neuroimaging. We explored the anatomical correlates of neglect patients' performance on distinct tests of neglect. For the first time in neglect anatomy studies, we individually assessed 25 patients with subacute strokes in the right hemisphere, by using a combined structural and diffusion tensor deterministic tractography approach, with separate analyses for each neglect test. The results revealed that lesions in nodes of the ventral attention network (angular and supramarginal gyri) were selectively associated with deficits in performance on all the tests used; damage to other structures correlated with impaired performance on specific tests, such as the bells test (middle and inferior frontal gyri), or the reading test (temporal regions). Importantly, however, white matter damage proved crucial in producing neglect-related deficits. Voxel-based lesion-symptom mapping (VLSM) and tractography consistently revealed that damage to the ventral branch of the superior longitudinal fasciculus (SLF III) and to the inferior fronto-occipital fasciculus (IFOF) predicted pathological scores on line bisection/drawing copy and on the bells test, respectively. Moreover, damage to distinct sectors of SLF III, or combined SLF/IFOF damage, gave rise to different performance profiles. Our results indicate that both grey and white matter lesion analysis must be taken into account to determine the neural correlates of neglect-related deficits. They also suggest that damage to distinct portions of white matter tracts may give rise to distinct clinical signs of neglect, presumably by inducing dysfunction of partly overlapping, but distinct networks.