Understanding the nature and extent of neural plasticity in humans remains a key challenge for neuroscience. Importantly, however, a precise characterization of plasticity and its underlying mechanism has the potential to enable new approaches for enhancing reorganization of cortical function. Investigations of the impairment and subsequent recovery of cognitive and perceptual functions following early-onset cortical lesions in humans provide a unique opportunity to elucidate how the brain changes, adapts, and reorganizes. Specifically, here, we focus on restitution of visual function, and we review the findings on plasticity and re-organization of the ventral occipital temporal cortex (VOTC) in published reports of 46 patients with a lesion to or resection of the visual cortex early in life. Findings reveal that a lesion to the VOTC results in a deficit that affects the visual recognition of more than one category of stimuli (faces, objects and words). In addition, the majority of pediatric patients show limited recovery over time, especially those in whom deficits in low-level vision also persist. Last, given that neither the equipotentiality nor the modularity view on plasticity was clearly supported, we suggest some intermediate possibilities in which some plasticity may be evident but that this might depend on the area that was affected, its maturational trajectory as well as its structural and functional connectivity constraints. Finally, we offer suggestions for future research that can elucidate plasticity further.