Attentional interference control is a prominent feature of human cognition. To what extent sexual stimuli attract attention and interfere with cognitive tasks has still little been studied. Our study aimed to identify associations between attentional interference, sexual arousal, trait sexual motivation, and neural activity to sexual distractors while accounting for gender differences. Therefore, the present study examined the neural correlates of attentional interference by arousing sexual distractors using functional magnetic resonance imaging (fMRI). Fifty women and 47 men underwent fMRI while indicating the orientation of two lines (equal or unequal) next to an explicit sexual (as compared to a neutral) picture. Results confirmed prolonged response times when a sexual image was shown. There was neither a difference between genders nor an effect of sexual arousal ratings or trait sexual motivation on distractibility. Neural activity specific to sexual images was found in brain regions implicated in motivation and reward processing. Men as compared to women showed stronger responses in the nucleus caudatus, the anterior cingulate cortex, and the nucleus accumbens. Trait sexual motivation was selectively correlated with nucleus caudatus activity. Taken together, findings support the notion that even when not in the focus, sexual images activate the brains' reward circuitry. Men's higher sensitivity to the rewarding value of sexual cues may be critical for their higher risk of addictive/compulsive sexual behaviors.