The auditory brain stem response to speech mimics the acoustic characteristics of the speech signal with remarkable fidelity. This makes it possible to derive from it considerable theoretical and clinically applicable information relevant to auditory processing of complex stimuli. Years of research have led to the current characterization of these neural events with respect to the underlying acoustic information they reflect. The majority of data reviewed here originates from studies using a /da/ syllable to elicit the brain stem response, which consists of transient and periodic (frequency following) neural activity. We describe how the human auditory brain stem response separately encodes source and filter characteristics of the acoustic signal, which reflect paralinguistic and linguistic information simultaneously conveyed in speech. In normal-hearing individuals, these two classes of response components (source and filter) are highly correlated within a class but not between classes. This response dissociation becomes pronounced when stimuli are presented in background noise or with faster stimulus rates. In addition, some learning-impaired children show a selective deficiency in the neural encoding of acoustic features associated with the filter characteristics of speech. These children show no deficits in the encoding of source components, further supporting the notion of separate neural mechanisms. Overall, the auditory brain stem response to speech provides a way to access subcortical auditory processing mechanisms and may be used as a biological marker of deficient sound encoding associated with learning and auditory processing disorders.