Several studies report sex differences in sensitivity to gendered stimuli. We assume many of these to reflect differences as to the sex to which one feels attracted rather than to biological sex per se. Investigating voice perception, a function of high social relevance, we show that the behavioural and neural (BOLD) responses to male and female voices are mediated by sex and sexual orientation. In heterosexual men and women, we found an opposite-sex effect, reflected in higher classification accuracy for and a response bias towards voices of the other sex, while the effect became apparent as same-sex effect in homosexual men and women. Overall, sexual orientation had a greater impact in women than in men and homosexual women were closer to men in their behavioural responses to female voices. The activation patterns were similar for hetero- and homosexual men, both groups showing increased activation in response to male compared to female voices in regions distributed across the temporo-parietal and insular cortex. In contrast, women had increased activation in response to voices of the desired sex. It appears that both sex and sexual orientation impact on a function as basal as voice perception. Our results underline the need to assess sexual orientation in study participants if conclusions on sex differences shall be drawn. Many of the reported sex differences in behaviour and brain function might be mediated by sexual orientation and we encourage further research into the interplay between sex and sexual orientation.