Research demonstrates an influence of gaze direction in emotion recognition. Here we examined whether facial affect similarly influences recognition of gaze direction. Across two studies we found that averted relative to direct gaze was processed more quickly and accurately when coupled with fear, and direct relative to averted gaze with anger. Also evident was that slower overall gaze processing was associated with increased interaction effects between emotion and gaze. Examining individual differences, therefore, enabled us to extend previous research examining speed of processing as a moderator of the interaction effect, while holding constant task demands and stimulus features. Unexpectedly, a main effect emerged such that averted relative to direct gaze was found to be processed more quickly and accurately overall. This effect was not moderated by processing speed and is discussed as a potential stimulus-driven effect that may help explain discrepant findings in the literature.