The role of the right hemisphere in recovery of language function after left hemisphere stroke remains unclear. This prospective study aimed to determine the relationship between bilateral hemispheric changes and severity of aphasia in patients with chronic left hemisphere stroke. Twenty-three patients with left hemisphere stroke with aphasia and 10 healthy control patients underwent high-resolution T1-weighted and diffusion tensor imaging. Patients with aphasia were assessed using the Korean version of the Western Aphasia Battery, and severity of aphasia was quantified in terms of aphasia quotient; patients were classified into two groups – the mild-to-moderate and severe groups – on the basis of severity of aphasia. Brain volume and fractional anisotropy in language-associated areas of the brain were determined by voxel-based morphometry analysis. Volume of white matter in the left hemisphere and fractional anisotropy of the bilateral pars opercularis, pars triangularis, angular gyrus, supramarginal gyrus, and superior longitudinal fasciculus were significantly lower in the severe aphasia group than in the control group. The findings of this study demonstrate the importance of right hemispheric integrity for recovery of language function in patients with chronic poststroke aphasia.