Male cichlid fish, Astatotilapia burtoni, live in a lek-like social system in shore pools of Lake Tanganyika, Africa, as one of two distinct social phenotypes: territorial (T) males that comprise approximately 10–30% of the population and nonterritorial (NT) males that make up the rest. T males are brightly colored either blue or yellow with chromatic body patterns and are larger, reproductively capable, and defend territories containing a food resource used to entice females to spawn with them. NT males are camouflage colored, smaller, have regressed gonads, and shoal with females. Importantly, males shift between these social states depending on their success in aggressive encounters. It is not known whether there is a difference between yellow and blue T morphs. Here we asked whether T males preferentially defend their territory against a male of the same or opposite color. T males observed in social groups had agonistic interactions predominantly with neighboring T males of the opposite color, and yellow morphs initiated significantly more aggressive interactions. When agonistic preference was tested experimentally, T males had significantly more agonistic interactions toward males of the opposite color, and yellow T males became territorial in the majority of those interactions. Taken together, these results suggest that male coloration is an important social signal among neighboring T males in this species and support the hypothesis that T males differentially direct agonistic behavior depending on the color of neighboring males.