Mental and neural representations of words are at the core of understanding the cognitive and neural mechanisms of reading. Despite extensive studies, the nature of visual word representation remains highly controversial due to methodological limitations. In particular, it is unclear whether the fusiform cortex contains only abstract orthographic representation, or represents both lower and higher level orthography as well as phonology. Using representational similarity analysis, we integrated behavioral ratings, computational models of reading and visual object recognition, and neuroimaging data to examine the nature of visual word representations in the fusiform cortex. Our results provided clear evidence that the middle and anterior fusiform represented both phonological and orthographic information. Whereas lower level orthographic information was represented at every stage of the ventral visual stream, abstract orthographic information was increasingly represented along the posterior-to-anterior axis. Furthermore, the left and right hemispheres were tuned to high- and low-frequency orthographic information, respectively. These results help to resolve the long-standing debates regarding the role of the fusiform in reading, and have significant implications for the development of psychological, neural, and computational theories of reading.