The ability to project oneself into the past and future to relive or pre-live personal experiences, known as mental time travel (MTT), is associated with activity in a core network of brain regions involving the ventromedial prefrontal cortex (vmPFC). We investigated whether (1) vmPFC is crucial for MTT, and (2) whether vmPFC is selectively involved in the construction of self-relevant events or also mediates construction of events happening to others. Patients with lesions to vmPFC (vmPFC patients) and healthy controls remembered personal past events and imagined personal future events across different timeframes, and imagined events to happen to a close or a distant other. Compared to the controls, vmPFC patients were impaired at constructing both past and future events, indicating that vmPFC is critical for MTT. vmPFC patients' ability to imagine personal future events was related to patients' temporal discounting rates. Patients, however, were also impaired at imagining other-related events, suggesting that self-relevance may not be a critical factor in explaining vmPFC's involvement in MTT. We suggest that vmPFC is crucial for the imagination of complex experiences alternative to the current reality, which serves construction of both self-relevant and other-relevant events.