It is suggested that the brain undergoes plastic changes in order to adapt to changing environmental needs. Sensory deprivation results in decreased input to the brain leading to adaptive or maladaptive changes. Although several theories hypothesize the mechanism of these adaptive and maladaptive changes, the course of action taken by the brain heavily depends on the age of incidence of damage. The growing body of literature on the topic proposes that maladaptive changes in the brain are instrumental in creating phantom percepts, defined as the perception of a sensory experience in the absence of a physical stimulus. The current article reviews the mechanisms of adaptive and maladaptive plasticity in the brain in congenital, early, and late-onset sensory deprivation in conjunction with the phantom percepts in the different sensory domains. We propose that the mechanisms of adaptive and maladaptive plasticity fall under a universal construct of updating hierarchical Bayesian prediction errors. This theory of the Bayesian brain hypothesizes that the brain constantly compares its internal milieu with changing environmental cues and either adjusts its predictions or discards the change, depending on the novelty or salience of the external stimulus. We propose that adaptive plasticity reflects both successful bottom-up compensation and top-down updating of the model while maladaptive plasticity reflects failure in one or both mechanisms, resulting in a constant prediction-error. Finally, we hypothesize that phantom percepts are generated by the brain as a solution to this prediction error and are thus a manifestation of unsuccessful adaptation to sensory deprivation.