The dominant paradigm for understanding working memory, or the combination of the perceptual, attentional, and mnemonic processes needed for thinking, subdivides short-term memory (STM) according to whether memoranda are encoded in aural-verbal or visual formats. This traditional dissociation has been supported by examples of neuropsychological patients who seem to selectively lack STM for either aural-verbal, visual, or spatial memoranda, and by experimental research using dual-task methods. Though this evidence is the foundation of assumptions of modular STM systems, the case it makes for a specialized visual STM system is surprisingly weak. I identify the key evidence supporting a distinct verbal STM system—patients with apparent selective damage to verbal STM and the resilience of verbal short-term memories to general dual-task interference—and apply these benchmarks to neuropsychological and experimental investigations of visual-spatial STM. Contrary to the evidence on verbal STM, patients with apparent visual or spatial STM deficits tend to experience a wide range of additional deficits, making it difficult to conclude that a distinct short-term store was damaged. Consistently with this, a meta-analysis of dual-task visual-spatial STM research shows that robust dual-task costs are consistently observed regardless of the domain or sensory code of the secondary task. Together, this evidence suggests that positing a specialized visual STM system is not necessary.