During dusk and dawn, the ambient illumination undergoes drastic changes in irradiance (or intensity) and spectrum (or color). While the former is a well-studied factor in synchronizing behavior and physiology to the earth’s 24-h rotation, color sensitivity in the regulation of circadian rhythms has not been systematically studied. Drawing on the concept of color opponency, a well-known property of image-forming vision in many vertebrates (including humans), we consider how the spectral shifts during twilight are encoded by a color-opponent sensory system for non-image-forming (NIF) visual functions, including phase shifting and melatonin suppression. We review electrophysiological evidence for color sensitivity in the pineal/parietal organs of fish, amphibians and reptiles, color coding in neurons in the circadian pacemaker in mice as well as sporadic evidence for color sensitivity in NIF visual functions in birds and mammals. Together, these studies suggest that color opponency may be an important modulator of light-driven physiological and behavioral responses.