Contrasting the impact of congenital versus late-onset acquired blindness provides a unique model to probe how experience at different developmental periods shapes the functional organization of the occipital cortex. We used functional magnetic resonance imaging to characterize brain activations of congenitally blind, late-onset blind and two groups of sighted control individuals while they processed either the pitch or the spatial attributes of sounds. Whereas both blind groups recruited occipital regions for sound processing, activity in bilateral cuneus was only apparent in the congenitally blind, highlighting the existence of region-specific critical periods for crossmodal plasticity. Most importantly, the preferential activation of the right dorsal stream (middle occipital gyrus and cuneus) for the spatial processing of sounds was only observed in the congenitally blind. This demonstrates that vision has to be lost during an early sensitive period in order to transfer its functional specialization for space processing toward a non-visual modality. We then used a combination of dynamic causal modelling with Bayesian model selection to demonstrate that auditory-driven activity in primary visual cortex is better explained by direct connections with primary auditory cortex in the congenitally blind whereas it relies more on feedback inputs from parietal regions in the late-onset blind group. Taken together, these results demonstrate the crucial role of the developmental period of visual deprivation in (re)shaping the functional architecture and the connectivity of the occipital cortex. Such findings are clinically important now that a growing number of medical interventions may restore vision after a period of visual deprivation.