This study investigated categorical perception for unique hues in order to establish a relationship between color appearance, color discrimination, and low-level (second-stage) mechanisms. We tested whether pure red, yellow, green, and blue, (unique hues) coincide with troughs, and their transitions (binary hues) with peaks of sensitivity in DKL-space. Results partially confirmed this idea: JNDs demarcated perceptual categories at the binary hues around green, blue and less clearly around yellow, when colors were isoluminant with the background and when accounting for the overall variation of sensitivity by fitting an ellipse. The categorical JND pattern for those three categories was in line with the effect of the second-stage mechanisms. In contrast, the results for unique red, binary red-yellow, and the JNDs for dark colors clearly contradicted categorical perception. There was a JND maximum around the center of red and JNDs strongly decreased away from the center. Although this observation alone would also be in line with categorical perception; unique red was shifted away from the center towards yellow so that unique red was close to the minimum instead of the maximum JND, hence contradicting categorical perception. In addition, we also showed that observers do not adjust unique hues more consistently than binary hues, confirming a previous study. Taken together, our findings suggest that some of the unique hues could be inherent in the early stages of color processing. At the same time, they also raise questions about complex effects of lightness, chroma and instructions on the measurements of JNDs and unique hues.