Studies using backward masked emotional stimuli suggest that affective processing may occur outside visual awareness and imply primacy of affective over semantic processing, yet these experiments have not strictly controlled for the participants’ awareness of the stimuli. Here we directly compared the primacy of affective versus semantic categorization of biologically relevant stimuli in 5 experiments (n = 178) using explicit (semantic and affective discrimination; Experiments 1–3) and implicit (semantic and affective priming; Experiments 4–5) measures. The same stimuli were used in semantic and affective tasks. Visual awareness was manipulated by varying exposure duration of the masked stimuli, and subjective level of stimulus awareness was measured after each trial using a 4-point perceptual awareness scale. When participants reported no awareness of the stimuli, semantic and affective categorization were at chance level and priming scores did not differ from zero. When participants were even partially aware of the stimuli, (a) both semantic and affective categorization could be performed above chance level with equal accuracy, (b) semantic categorization was faster than affective categorization, and (c) both semantic and affective priming were observed. Affective categorization speed was linearly dependent on semantic categorization speed, suggesting dependence of affective processing on semantic recognition. Manipulations of affective and semantic categorization tasks revealed a hierarchy of categorization operations beginning with basic-level semantic categorization and ending with superordinate level affective categorization. We conclude that both implicit and explicit affective and semantic categorization is dependent on visual awareness, and that affective recognition follows semantic categorization.