Segmenting word units from running speech is a fundamental skill infants must develop in order to acquire language. Despite ample behavioral evidence of this skill, its neurocognitive basis remains unclear. Using behavioral testing and functional near-infrared spectroscopy, we aimed to uncover the neurocognitive substrates of word segmentation and its development. Of three age-groups of Japanese infants (5–6, 7–8, and 9–10 months of age), the two older age-groups showed significantly larger temporo-parietal (particularly supramarginal gyrus) responses to target words repeatedly presented for training, than to control words. After the training, they also exhibited stronger inferior frontal responses to target words embedded in sentences. These findings suggest that word segmentation largely involves a cerebral circuit of phonological (phonetic) short-term memory. The dorsal pathway involved in encoding and decoding phonological representation may start to function stably at around 7 months of age to facilitate the growth of the infant’s vocabulary.