The neurobiological alterations resulting from adverse childhood experiences that subsequently may lead to neglectful mothering are poorly understood. Maternal neglect of an infant’s basic needs is the most prevalent type of child maltreatment. We tested white matter alterations in neglectful mothers, the majority of whom had also suffered maltreatment in their childhood, and compared them to a matched control group. The two groups were discriminated by a structural brain connectivity pattern comprising inferior fronto-temporo-occipital connectivity, which constitutes a major portion of the face-processing network and was indexed by fewer streamlines in neglectful mothers. Mediation and regression analyses showed that fewer streamlines in the right inferior longitudinal fasciculus tract (ILF-R) predicted a poorer quality of mother–child emotional availability observed during cooperative play and that effect depended on the respective interactions with left and right inferior fronto-occipital fasciculi (IFO-R/L), with no significant impact of psychopathological and cognitive conditions. Volume alteration in ILF-R but not in IFO-L modulated the impact of having been maltreated on emotional availability. The findings suggest the altered inferior fronto-temporal-occipital connectivity, affecting emotional visual processing, as a possible common neurological substrate linking a history of childhood maltreatment with maternal neglect.