Visual cortical neurons are sensitive to visual stimulus contrast and most cells adapt their sensitivity to the prevailing visual environment. Specifically, they match the steepest region of their contrast response function to the prevailing contrast (contrast gain control), and reduce spike rates to limit saturation (response gain control). Most neurons are also tuned for stimulus orientation, and neurons with similar orientation preference are clustered together into iso-orientation zones arranged around pinwheels, i.e. points where all orientations are represented. Here we investigated the relationship between the contrast adaptation properties of neurons and their location relative to pinwheels in the orientation preference map. We measured orientation preference maps in cat cortex using optical intrinsic signal imaging. We then characterized the contrast adaptation properties of single neurons located close to pinwheels, in iso-orientation zones, and at regions in between. We found little evidence of differential contrast sensitivity of neurons adapted to zero contrast. However, after adaptation to their preferred orientation at high contrast, changes in both contrast and response gain were greater for neurons near pinwheels compared with other map regions. Therefore, in the adapted state, which is probably typical during natural viewing, there is a spatial map of contrast sensitivity that is associated with the orientation preference map. This differential adaptation revealed a new dimension of cortical functional organization, linking the contrast adaptation of cells with the orientation preference of their nearest neighbours.