Emotional empathy—feeling another person’s affective states—entails simulating how one would feel in the same circumstance. Prior research has implicated the role of executive controls and shown a link between visuospatial perspective taking and personal disposition of empathy. No study has investigated how executive control processes involved in perspective shifting relate to emotional empathy. Incorporating a spatial perspective-taking task in a set switch paradigm, we investigated whether swiftly switching from the altercentric to the egocentric perspective is associated with heightened emotional empathy but not with accurate classification of low-level perceptual affective cues. Emotional empathy was measured by subjective ratings of arousal and the similarity of affective states with the target person when viewing photos of a person in an emotionally charged context. Cognitive empathy was measured by correct recognition of affective cues. Our results showed that executive controls in perspective shifting related to emotional empathy but not to cognitive empathy. Emotional empathy correlated negatively with the switch cost from the altercentric to the egocentric perspective and not vice versa. Faster switching from the altercentric to the egocentric perspective was associated with heightened emotional empathy. Moreover, the processing strategy did not moderate the association. Flexibility in perspective shifting, especially in regaining one’s own perspective after taking another person’s perspective, is critical for emotional empathy. To feel another person’s affective states, one should regain self-perspective after walking in the other person’s shoes.