Attention is critical to the construction of mental representations of language context during comprehension. We investigated the consequences of momentary lapses in attention during listening comprehension on neural activity and behavior. Participants listened to two full-length stories while EEG was recorded, and afterwards completed multiple choice comprehension questions. Listening was periodically interrupted by attention probes, in which participants were asked whether their attention immediately preceding the probe's appearance was focused on the story. The results showed that (1) participants spent a substantial amount of time off-task, endorsing attention lapses on over 30% of probes; (2) for probes on which an attention lapse was endorsed, later accuracy on comprehension questions querying pre-probe information was decreased; (3) the pre-probe period just before the endorsement of an attention lapse was characterized by a greater percentage of above-threshold oscillations in the alpha-band (8–12 Hz) compared to just prior to the endorsement of on-task or split-attention listening; and (4) when participants made “I have no idea” responses to comprehension questions, their EEG record revealed a greater percentage of above-threshold alpha oscillations during the original presentation of the information queried by the comprehension questions, compared to correct responses or incorrect guesses. These results connect changes in neural activity in the alpha band to episodes of mind-wandering during listening comprehension, and in turn to decreased comprehension accuracy. This demonstrates how alpha can be used to track attentional engagement during language comprehension, and illustrates the dependence of successful language comprehension on attention.