In highly social groups like human and non-human primates, gaze and pointing cues are fundamentally important for directing the attention of conspecifics. Although neuroimaging studies indicate that shifts of attention triggered by observation of social cues activate the onlookers' fronto-parietal cortices, information on whether these regions play a causative role in orienting and re-orienting of social attention is lacking. To advance our understanding of this, we used event-related repetitive dual pulse transcranial magnetic stimulation to interfere with neural activity in the right frontal eye field (rFEF) and posterior parietal cortex (rPPC). This procedure allowed us to explore how inhibiting rFEF and rPPC influences shifts of attention triggered by the observation of body-related (gaze and hand) and non body-related (arrow) directional distractors. Participants were asked to perform a leftward or rightward pointing movement according to the color change of a central imperative signal while ignoring a distractor, which was either a gaze, a pointing hand or an arrow. Stimulation of rPPC in a region supposedly linked to attentional re-orienting and to planning and execution of upper limb movements increased the reflexive tendency to follow distracting pointing hands but not oriented gaze or arrows. These findings suggest that inhibition of cortical structures that control attentional shifts triggered by social stimuli brings forth an increase of the cost of attentional re-orienting. Moreover, our results provide the first causative evidence that reflexive social attention in humans may be coded according to body-part-centered frames of reference.