Adaptive human behavior crucially relies on the ability of the brain to allocate resources automatically to emotionally significant stimuli. This ability has consistently been demonstrated by studies showing preferential processing of affective stimuli in sensory cortical areas. It is still unclear, however, whether this putatively automatic mechanism can be modulated by cognitive control processes. Here, we use functional magnetic resonance imaging (fMRI) to investigate whether preferential processing of an affective face distractor is suppressed when an affective distractor has previously elicited a response conflict in a word-face Stroop task. We analyzed this for three consecutive stages in the ventral stream of visual processing for which preferential processing of affective stimuli has previously been demonstrated: the striate area (BA 17), category-unspecific extrastriate areas (BA 18/19), and the fusiform face area (FFA). We found that response conflict led to a selective suppression of affective face processing in category-unspecific extrastriate areas and the FFA, and this effect was accompanied by changes in functional connectivity between these areas and the rostral anterior cingulate cortex. In contrast, preferential processing of affective face distractors was unaffected in the striate area. Our results indicate that cognitive control processes adaptively suppress preferential processing of affective stimuli under conditions where affective processing is detrimental because it elicits response conflict.