Functional imaging studies suggest that accurate understanding of others’ emotional feelings (i.e., empathic accuracy, EA) recruits high-order visual, sensorimotor and mentalizing brain networks. However, the behavioral relevance of these findings is unclear. To fill in this gap, we used repetitive transcranial magnetic stimulation (rTMS) to interfere with the right superior temporal sulcus (STS), inferior frontal gyrus (IFG) and temporoparietal junction (TPJ) during an EA task requiring participants to infer the enjoyment felt by a social target while smiling/laughing. Relative to a baseline condition (sham rTMS), active rTMS of STS, IFG and TPJ (but not of a control site) disrupted the efficiency of EA task performance, mainly by lowering task accuracy; rTMS of IFG and TPJ also slowed down response speeds. Importantly, the effects of rTMS on EA task efficiency were predicted by baseline EA performance, with high-performers showing a performance decrease when the TPJ was targeted, and low-performers showing a performance decrease when the STS or the IFG was targeted. The double dissociation in the effect of rTMS between low- and high-performers suggests distinct roles of STS, IFG and TPJ in efficient understanding of the enjoyment felt by others. These findings provide causal evidence of distinct visual, sensorimotor and cognitive routes to EA and suggest that individual differences in EA are underpinned by differential recruitment of these routes.