Modality compatibility refers to the similarity of stimulus modality and modality of response-related sensory consequences. Previous dual-task studies found increased switch costs for modality incompatible tasks (auditory-manual/visual-vocal) compared to modality compatible tasks (auditory-vocal/visual-manual). The present task-switching study further examined modality compatibility and investigated vibrotactile stimulation as a novel alternative to visual stimulation. Interestingly, a stronger modality compatibility effect on switch costs was revealed for the group with tactile-auditory stimulation compared to the visual-auditory stimulation group. We suggest that the modality compatibility effect is based on crosstalk of central processing codes due to ideomotor “backward” linkages between the anticipated response effects and the stimuli indicating this response. This crosstalk is increased in the tactile-auditory stimulus group compared to the visual-auditory stimulus group due to a higher degree of ideomotor-compatibility in the tactile-manual tasks. Since crosstalk arises between tasks, performance is only affected in task switching and not in single tasks.