Women typically remember more female than male faces, whereas men do not show a reliable own-gender bias. However, little is known about the neural correlates of this own-gender bias in face recognition memory. Using functional magnetic resonance imaging (fMRI), we investigated whether face gender modulated brain activity in fusiform and inferior occipital gyri during incidental encoding of faces. Fifteen women and 14 men underwent fMRI while passively viewing female and male faces, followed by a surprise face recognition task. Women recognized more female than male faces and showed higher activity to female than male faces in individually defined regions of fusiform and inferior occipital gyri. In contrast, men’s recognition memory and blood-oxygen-level-dependent response were not modulated by face gender. Importantly, higher activity in the left fusiform gyrus (FFG) to one gender was related to better memory performance for that gender. These findings suggest that the FFG is involved in the gender bias in memory for faces, which may be linked to differential experience with female and male faces.