Associations between genotypes and cognitive outcomes may provide clues as to which mechanisms cause individual differences in old-age cognitive performance. We investigated the effects of five polymorphisms on cognitive functioning in a population-based sample of 2,694 persons without dementia (60–102 years). A structural equation model (SEM) was fit to the cognitive data, yielding five specific latent factors (perceptual speed, episodic memory, semantic memory, category fluency, and letter fluency), as well as a global cognitive factor. These factors showed the expected associations with chronological age. Genotyping was performed for five single-nucleotide polymorphisms that have been associated with cognitive performance: APOE (rs429358), COMT (rs4680), BDNF (rs6265), KIBRA (rs17070145), and CLSTN2 (rs6439886). After controlling for age, gender, and education, as well as correcting for multiple comparisons, we observed negative effects of being an APOE ε4 carrier on episodic memory and perceptual speed. Furthermore, being a CLSTN2 TT carrier was associated with poorer semantic memory. For the global factor, the same pattern of results was observed. In addition, being a BDNF any A carrier was associated with better cognitive performance. Also, older age was associated with stronger genetic effects of APOE on global cognition. However, this interaction effect was partly driven by the presence of preclinical dementia cases in our sample. Similarly, excluding future dementia cases attenuated the effects of APOE on episodic memory and global cognition, suggesting that part of the effects of APOE on old-age cognitive performance may be driven by dementia-related processes.