Pre- or perinatally acquired (“congenital”) left-hemispheric brain lesions can be compensated for by reorganizing language into homotopic brain regions in the right hemisphere. Language comprehension may be hemispherically dissociated from language production. We investigated the lesion characteristics driving inter-hemispheric reorganization of language comprehension and language production in 19 patients (7–32 years; eight females) with congenital left-hemispheric brain lesions (periventricular lesions [n = 11] and middle cerebral artery infarctions [n = 8]) by fMRI. 16/17 patients demonstrated reorganized language production, while 7/19 patients had reorganized language comprehension. Lesions to the insular cortex and the temporo-parietal junction (predominantly supramarginal gyrus) were significantly more common in patients in whom both, language production and comprehension were reorganized. These areas belong to the dorsal stream of the language network, participating in the auditory-motor integration of language. Our data suggest that the integrity of this stream might be crucial for a normal left-lateralized language development.