Currently, a distributed bilateral network of frontal-parietal areas is regarded as the neural substrate of working memory (WM), with the verbal WM network being more left-lateralized. This conclusion is based primarily on functional magnetic resonance imaging (fMRI) data that provides correlational evidence for brain regions involved in a task. However, fMRI cannot differentiate the areas that are fundamentally required for performing a task. These data can only come from brain-injured individuals who fail the task after the loss of specific brain areas. In addition to the lack of complimentary data, is the issue of the variety in the WM tasks used to assess verbal WM. When different tasks are assumed to measure the same behavior, this may mask the contributions of different brain regions. Here, we investigated the neural substrate of WM by using voxel-based lesion symptom mapping (VLSM) in 49 individuals with stroke-induced left hemisphere brain injuries. These participants completed two different verbal WM tasks: complex listening span and a word 2-back task. Behavioral results indicated that the two tasks were only slightly related, while the VLSM analysis revealed different critical regions associated with each task. Specifically, significant detriments in performance on the complex span task were found with lesions in the inferior frontal gyrus, while for the 2-back task, significant deficits were seen after injury to the superior and middle temporal gyri. Thus, the two tasks depend on the structural integrity of different, non-overlapping frontal and temporal brain regions, suggesting distinct neural and cognitive mechanisms triggered by the two tasks: rehearsal and cue-dependent selection in the complex span task, versus updating/auditory recognition in the 2-back task. These findings call into question the common practice of using these two tasks interchangeably in verbal WM research and undermine the legitimacy of aggregating data from studies with different WM tasks. Thus, the present study points out the importance of lesion studies in complementing functional neuroimaging findings and highlights the need to consider task demands in neuroimaging and neuropsychological investigations of WM.