Alexithymia refers to deficiencies in identifying and expressing emotions. This might be related to changes in structural brain volumes, but its neuroanatomical basis remains uncertain as studies have shown heterogeneous findings. Therefore, we conducted a parametric coordinate-based meta-analysis. We identified seventeen structural neuroimaging studies (including a total of 2586 individuals with different levels of alexithymia) investigating the association between gray matter volume and alexithymia. Volumes of the left insula, left amygdala, orbital frontal cortex and striatum were consistently smaller in people with high levels of alexithymia. These areas are important for emotion perception and emotional experience. Smaller volumes in these areas might lead to deficiencies in appropriately identifying and expressing emotions. These findings provide the first quantitative integration of results pertaining to the structural neuroanatomical basis of alexithymia.