Social interactions depend on mechanisms such as the ability to take another person's viewpoint, i.e. visuo-spatial perspective taking. However, little is known about the sensorimotor mechanisms underpinning perspective taking. Because vestibular signals play roles in mental rotation and spatial cognition tasks and because damage to the vestibular cortex can disturb egocentric perspective, vestibular signals stand as important candidates for the sensorimotor foundations of perspective taking. Yet, no study merged natural full-body vestibular stimulations and explicit visuo-spatial perspective taking tasks in virtual environments. In Experiment 1, we combined natural vestibular stimulation on a rotatory chair with virtual reality to test how vestibular signals are processed to simulate the viewpoint of a distant avatar. While they were rotated, participants tossed a ball to a virtual character from the viewpoint of a distant avatar. Our results showed that vestibular signals influence perspective taking in a direction-specific way: participants were faster when their physical body rotated in the same direction as the mental rotation needed to take the avatar's viewpoint. In Experiment 2, participants realized 3D object mental rotations, which did not involve perspective taking, during the same whole-body vestibular stimulation. Our results demonstrated that vestibular stimulation did not affect 3D object mental rotations. Altogether, these data indicate that vestibular signals have a direction-specific influence on visuo-spatial perspective taking (self-centered mental imagery), but not a general effect on mental imagery. Findings from this study suggest that vestibular signals contribute to one of the most crucial mechanisms of social cognition: understanding others' actions.