Functional neuroimaging studies of numerical cognition have repeatedly associated activation of the intraparietal sulcus (IPS) with number processing. During number comparison, the IPS has been found to be modulated by the numerical distance. This has lead to the contention that the IPS houses the internal representation of numerical magnitude. However, this theory has been challenged by the argument that IPS activation may reflect domain-general response selection. In the present study, we used the numerical size congruity paradigm to further elucidate the role played by the IPS in number comparison. In an event-related, functional magnetic resonance imaging (fMRI) study, participants judged which of two number words was numerically larger. In addition to the numerical distance, physical stimulus size was varied such that physical size and numerical magnitude were either (a) congruent (e.g., numerically smaller number printed in smaller font) or (b) incongruent (e.g., numerically larger number printed in smaller font). This allowed for the study of both the main effects and the interaction of numerical distance and stimulus congruency. A main effect of numerical distance was found in bilateral regions of the IPS. However, these parietal areas were not significantly modulated by congruency or the interaction of distance and congruency. Instead, the main effect of congruency and an interaction of distance and congruency were observed in anterior cingulate and prefrontal cortices. These findings suggest some degree of independence between the processing of numerical distance and size congruity, lending support for the hypothesis that distance effects in IPS reflect the underlying representation of numerical magnitude.