We use an fMRI adaptation paradigm to explore the selectivity of human responses in the lateral geniculate nucleus (LGN) and superior colliculus (SC) to red–green color and achromatic contrast. We measured responses to red–green (RG) and achromatic (ACH) high contrast sinewave counter-phasing rings with and without adaptation, within a block design. The signal for the RG test stimulus was reduced following both RG and ACH adaptation, whereas the signal for the ACH test was unaffected by either adaptor. These results provide compelling evidence that the human LGN and SC have significant capacity for color adaptation. Since in the LGN red–green responses are mediated by P cells, these findings are in contrast to earlier neurophysiological data from non-human primates that have shown weak or no contrast adaptation in the P pathway. Cross-adaptation of the red–green color response by achromatic contrast suggests unselective response adaptation and points to a dual role for P cells in responding to both color and achromatic contrast. We further show that subcortical adaptation is not restricted to the geniculostriate system, but is also present in the superior colliculus (SC), an oculomotor region that until recently, has been thought to be color-blind. Our data show that the human SC not only responds to red–green color contrast, but like the LGN, shows reliable but unselective adaptation.