Grapheme-color synesthesia is a phenomenon in which achromatic letters/digits automatically induce particular colors. When multiple letters are integrated into a word, some synesthetes perceive that all those letters are changed into the same color, reporting lexical color to that word. Previous psychological studies found several “rules” that determine those lexical colors. The colors to most words are determined by the first letters of the words, while some words in ordinal sequences have their specific colors. Recent studies further reported the third case where lexical colors might be influenced by semantic information of words. Although neural mechanisms determining those lexical colors remained unknown, here we identified three separate neural systems in the synesthete's brain underlying three rules for illusory coloring of words. In addition to the occipito-temporal and parietal regions previously found to be associated with the grapheme-color synesthesia, neural systems for lexical coloring extended to linguistic areas in the left inferior frontal and anterior temporal regions that were engaged in semantic analyses of words. Those results indicate an involvement of wider and higher neural networks than previously assumed in a production of synesthetic colors to visual stimuli and further showed a multiplicity of synesthetic mechanisms represented in the single brain.