The purpose of the current review is to highlight the role of the acoustic environment in auditory cortical plasticity. In order do this we have reviewed our past studies on auditory cortical plasticity based on long-latency evoked potential recordings in humans following cochlear implantation, and multiple single-unit recordings from cat auditory cortex following noise trauma and exposure to a non-deafening acoustic environment. The results of these studies, and those of other investigators highlighted here, show that the auditory cortex shows plastic changes throughout life. Those that occur during maturation are typically considered the most profound and long lasting. In that case plasticity is beneficial as it allows adaptation to behaviorally important sound and adapts easily to changes induced by deafness and subsequent application of hearing aids or cochlear implants. In children as well as adults, changes in cortical representation of frequency can occur following hearing loss, but may be accompanied by unpleasant side effects such as tinnitus. Long exposure to a spectrally enhanced acoustic environment of moderate sound level that does not cause hearing loss paradoxically also results in pronounced changes in the cortical tonotopic maps. These changes are very similar to those following noise trauma. This review provides evidence that in adults, long-lasting plastic changes in auditory cortex occur even in the absence of behaviorally relevant acoustic stimulation. However, in children, the long lasting absence of auditory stimulation arrests cortical development.