The extent to which scenes are categorized and understood in the absence of attention has been the focus of a continuous debate over the last decade. Most studies investigating this question have used experimental paradigms in which participants explicitly searched for a certain scene category, or alternatively, scenes were task-irrelevant yet their identity was explicitly reported by participants. Although the first type of studies may have overestimated unattended scene processing, the latter type of studies may have underestimated scene processing due to the reliance on subjective response criteria and on working memory capacity limits. The present research examined scene processing by using an implicit, online behavioral measure which assessed the influence of both task-relevant (i.e., to-be-detected) and task-irrelevant distractor scenes on behavior. The effect of scene categorization was compared when scenes were fully attended (Experiment 1) versus when they were positioned in an unattended location and served as relevant/irrelevant distractors (Experiments 2 and 3). Our results demonstrated that in contrast to attended scenes, unattended distractor scenes which were not part of one’s task-set were not automatically categorized and did not exert influence on performance. Critically, however, the very same scene distractors affected behavior when they contained a to-be-detected category, suggesting a qualitative dissociation between task-relevant and task-irrelevant distractors. Our study provides a systematic examination of scene distractor processing outside the focus of visual attention and a framework that may reconcile previous conflicting evidence.