In the present study, we searched for causal evidence linking activity in the bilateral primary somatosensory cortex (SI), posterior parietal cortex (PPC), and prefrontal cortex (PFC) with behavioral performance in vibrotactile working memory. Participants performed a vibrotactile delayed matching-to-sample task, while single-pulse transcranial magnetic stimulation (sp-TMS) was applied over these cortical areas at 100, 200, 300, 600, 1600, and 1900 ms after the onset of vibrotactile stimulation (200 ms duration). In our experiments, sp-TMS over the contralateral SI at the early delay (100 and 200 ms) deteriorated the accuracy of task performance, and over the ipsilateral SI at the late delay (1600 and 1900 ms) also induced such deteriorating effects. Furthermore, deteriorating effects caused by sp-TMS over the contralateral DLPFC at the same maintenance stage (1600 ms) were correlated with the effects caused by sp-TMS over the ipsilateral SI, indicating that information retained in the ipsilateral SI during the late delay may be associated with the DLPFC. Taken together, these results suggest that both the contralateral and ipsilateral SIs are involved in tactile WM, and the contralateral DLPFC bridges the contralateral SI and ipsilateral SI for goal-directed action.