Most men have a category-specific pattern of genital and subjective sexual arousal, responding much more strongly to erotic stimuli depicting their preferred sex than to erotic stimuli depicting their nonpreferred sex. In contrast, women tend to have a less specific arousal pattern. To better understand this sex difference, we used neuroimaging to explore its neural correlates. Heterosexual and homosexual women viewed erotic photographs of either men or women. Evoked neural activity was monitored via fMRI and compared with responses to the same stimuli in heterosexual and homosexual men. Overall, a network of limbic (as well as the anterior cingulate) and visual processing regions showed significantly less category-specific activity in women than men. This was primarily driven by weaker overall activations to preferred-sex stimuli in women, though there was also some evidence of stronger limbic activations to nonpreferred-sex stimuli in women. Primary results were similar for heterosexual and homosexual participants. Women did show some evidence of category-specific responses in the visual processing regions, although even in these regions they exhibited less differential activity than men. In the anterior cingulate, a region with high concentrations of sex-hormone receptors, subjective and neural category specificity measures correlated positively for women but negatively for men, suggesting a possible sex difference in the role of the anterior cingulate. Overall, results suggest that men tend to show more differentiated neural responses than do women to erotic photographs of one sex compared to the other sex, though women may not be entirely indifferent to which sex is depicted.