Despite mental imagery's ubiquitous role in human perception, cognition and behavior, one standout question remains unanswered: Why does imagery vary so much from one individual to the next? Here, we used a behavioral paradigm that measures the functional impact of a mental image on subsequent conscious perception and related these measures to the anatomy of the early visual cortex estimated by fMRI retinotopic mapping. We observed a negative relationship between primary visual cortex (V1) surface area and sensory imagery strength, but found positive relationships between V1 and imagery precision (spatial location and orientation). Hence, individuals with a smaller V1 tended to have stronger, but less precise imagery. In addition, subjective vividness of imagery was positively related to prefrontal cortex volume, but unrelated to V1 anatomy. Our findings present the first evidence for the importance of the V1 layout in shaping the strength of human imagination.