Sex stereotypes consider women to be superior in terms of voice sensitivity. However, whether such sex differences are driven by voice perception per se or by low-level acoustic attributes remains unclear. Using a passive auditory oddball paradigm, we studied the emotionally spoken meaningless syllables ‘dada’ (neutral, happy, fearful) along with corresponding nonvocal sounds in female and male adults. Mismatch negativity (MMN) and P3a were identified in the waveforms obtained by subtraction of the responses to standard stimuli from the deviant stimuli. Results showed that MMN in response to fearful syllables was stronger in women, whereas MMN elicited by nonvocal sounds was comparable between sexes. This sex effect was specifically found in MMN, but not in P3a. These findings suggest that sex differences in voice sensitivity are selectively driven by voice perception. The sex effect in preattentive processing of emotional voices may further implicate possible etiological pathways for mental disorders characterized by disturbance in emotional processes as well as disparities in the prevalence between sexes.