The study of developmental disorders can provide a unique window into the role of domain-general cognitive abilities and neural systems in typical and atypical development. Mathematical disabilities (MD) are characterized by marked difficulty in mathematical cognition in the presence of preserved intelligence and verbal ability. Although studies of MD have most often focused on the role of core deficits in numerical processing, domain-general cognitive abilities, in particular working memory (WM), have also been implicated. Here we identify specific WM components that are impaired in children with MD and then examine their role in arithmetic problem solving. Compared to typically developing (TD) children, the MD group demonstrated lower arithmetic performance and lower visuo-spatial working memory (VSWM) scores with preserved abilities on the phonological and central executive components of WM. Whole brain analysis revealed that, during arithmetic problem solving, left posterior parietal cortex, bilateral dorsolateral and ventrolateral prefrontal cortex, cingulate gyrus and precuneus, and fusiform gyrus responses were positively correlated with VSWM ability in TD children, but not in the MD group. Additional analyses using a priori posterior parietal cortex regions previously implicated in WM tasks, demonstrated a convergent pattern of results during arithmetic problem solving. These results suggest that MD is characterized by a common locus of arithmetic and VSWM deficits at both the cognitive and functional neuroanatomical levels. Unlike TD children, children with MD do not use VSWM resources appropriately during arithmetic problem solving. This work advances our understanding of VSWM as an important domain-general cognitive process in both typical and atypical mathematical skill development.