Reductions in sound-evoked activity in the auditory nerve due to hearing loss have been shown to cause pathological changes in central auditory structures. Hearing loss due strictly to the aging process are less well documented. In this study of CBA/CaH mice, we provide evidence for age-related pathology in the endbulb of Held, a large axosomatic ending arising from myelinated auditory nerve fibers. Endbulbs are known to be involved in the processing of temporal cues used for sound localization and speech comprehension. Hearing thresholds as measured by auditory brainstem response (ABR) thresholds remained stable up to one year, whereas suprathreshold amplitudes of early ABR waves decreased by up to 50% in older mice, similar to that reported for age-related cochlear synaptopathy (Sergeyenko et al., 2013). The reduction of ABR response magnitude with age correlated closely in time with the gradual atrophy of endbulbs of Held, and is consistent with the hypothesis that endbulb integrity is dependent upon normal levels of spike activity in the auditory nerve. These results indicate that central auditory pathologies emerge as consequence of so-called “hidden” hearing loss and suggest that such brain changes require consideration when devising therapeutic interventions.