Few studies have examined the association between ambient temperature and cognitive function, or used exposure to temperature at a given address instead of a single stationary monitor. The existing literature on the temperature-cognition relationship has mostly consisted of experimental studies that involve a small sample size and a few specific temperature values. In the current study, we examined the association between residential air temperature and Mini-Mental State Examination (MMSE) scores, a quantitative measurement of cognitive function, in a longitudinal cohort of elderly men. Residential air temperature was estimated by a novel spatiotemporal approach that incorporates satellite remote sensing, land use regression, meteorological variables and spatial smoothing in the Northeastern USA. We then applied logistic regression generalized estimating equations to examine the relationship between residential temperature (range: −5.8–25.7 °C), and the risk of low MMSE scores (MMSE scores ≤25) among 594 elderly men (1085 visits in total) from the Veterans Affairs Normative Aging Study, 2000–2008. Sensitivity analysis on visits wherein subjects lived within 30 km of the clinic center in Massachusetts or aged ≥70 years was also evaluated. A statistically significant, U-shaped association between residential air temperature and low MMSE score (p-value=0.036) was observed. Sensitivity analysis suggested that the estimated effect remains among individuals aged ≥70 years. In conclusion, the data suggest that risk of low MMSE scores is highest when temperature is either high or low, and lowest when ambient temperature is approximately within 10–15 °C in a cohort of elderly men. Further research is needed to confirm our findings and assess generalizability to other populations.