The speed of recognizing facial expressions of emotion is influenced by a range of factors including other concurrently present facial attributes, like a person’s sex. Typically, when participants categorize happy and angry expressions on male and female faces, they are faster to categorize happy than angry expressions displayed by females, but not displayed by males. Using the same emotional faces across tasks, we demonstrate that this influence of sex cues on emotion categorization is dependent on the other faces recently encountered in an experiment. Altering the salience of gender by presenting male and female faces in separate emotion categorization tasks rather than together in a single task changed the influence of sex cues on emotion categorization, whereas changing the evaluative dimension by presenting happy and angry expressions in separate tasks alongside neutral faces rather than together within 1 task did not. These results suggest that the way facial attributes influence emotion categorization depends on the situation in which the faces are encountered and specifically on what information is made salient within or across tasks by other recently encountered faces.