Recognizing and understanding the actions of others is usually coupled with perceiving someone else's body movements from a third person perspective (3pp) whereas we perceive our own actions from a first person perspective (1pp). From a neural viewpoint, a recent finding is that perceiving actions from a 3pp as compared to a 1pp activates the temporoparietal junction, a brain region associated with visuospatial transformation and perspective taking but also with mental state inference and Theory of Mind (ToM). The present fMRI study characterizes the response profile of TPJ to elucidate its role in action observation. Participants observed naturalistic and pixelized object-directed actions from a 3pp and 1pp. Critically, in the pixelized condition the action goal could only be inferred from the movement kinematics. Both left and right TPJ revealed an interaction: Neural activity in TPJ was enhanced for 3pp vs. 1pp actions in the naturalistic but not pixelized condition. This finding contradicts theories proposing that TPJ is generally involved in transforming the action into the observer's perspective to match perceived body movements with visuomotor representations in the observer's motor system, which would be particularly required when actions can only be inferred from movement kinematics. Instead, our results support the theory that perceptual 3pp-selective cues trigger ToM-related processes such as detection of other agents and reasoning about an action's underlying mental states.