This study investigated the effects of early- to midlife musical training on cognition in older adults. A musical training survey examined self-reported musical experience and objective knowledge in 237 cognitively intact participants. Responses were classified into low-, medium-, and high-knowledge groups. Linear mixed models compared the groups’ longitudinal performance on the Animal Naming Test (ANT; semantic verbal fluency) and Logical Memory Story A Immediate Recall (LMI; episodic memory) controlling for baseline age, time since baseline, education, sex, and full-scale IQ. Results indicate that high-knowledge participants had significantly higher LMI scores at baseline and over time compared to low-knowledge participants. The ANT scores did not differ among the groups. Ability to read music was associated with higher mean scores for both ANT and LMI over time. Early- to midlife musical training may be associated with improved late-life episodic and semantic memory as well as a useful marker of cognitive reserve.