High-level regions of the ventral stream exhibit strong category selectivity to stimuli such as faces, houses, or objects. However, recent studies suggest that at least part of this selectivity stems from low-level differences inherent to images of the different categories. For example, visual outdoor and indoor scenes as well as houses differ in spatial frequency, rectilinearity and obliqueness when compared to face or object images. Correspondingly, scene responsive para-hippocampal place area (PPA) showed strong preference to low-level properties of visual scenes also in the absence of high-level scene content. This raises the question whether all high-level responses in PPA, the fusiform face area (FFA), or the object-responsive lateral occipital compex (LOC) may actually be explained by systematic differences in low-level features. In the present study we contrasted two classes of simple stimuli consisting of ten rectangles each. While both were matched in visual low-level features only one class of rectangle arrangements gave rise to a percept compatible with a high-level 3D layout such as a scene or an object. We found that areas PPA, transverse occipital sulcus (TOS, also referred to as occipital place area, OPA), as well as FFA and LOC showed robust responses to the visual scene class compared to the low-level matched control. Our results suggest that visual category responsive regions are not purely driven by low-level visual features but also by the high-level perceptual stimulus interpretation.