Since the earliest attempts to characterize the “receptive fields” of neurons, a central aim of many neuroscience experiments is to elucidate the information that is represented in various regions of the brain. Recent studies suggest that, in the service of memory, information is represented in the medial temporal lobe in a conjunctive or associative form with the contextual aspects of the experience being the primary factor or highest level of the conjunctive hierarchy. A critical question is whether the information that has been observed in these studies reflects notions such as a cognitive representation of context or whether the information reflects the low-level sensory differences between stimuli. We performed two functional magnetic resonance imaging experiments to address this question and we found that associative representations observed between context and item (and order) in the human brain can be highly influenced by low-level sensory differences between stimuli. Our results place clear constraints on the experimental design of studies that aim to investigate the representation of contexts and items during performance of associative memory tasks. Moreover, our results raise interesting theoretical questions regarding the disambiguation of memory-related representations from processing-related representations.