The risk of dementia and Alzheimer's disease (AD) probably results from an interaction between genetic and environmental factors. The aim of this study was to investigate the effects and putative interactions between the apoE ε4 allele and lifestyle related risk factors for dementia and AD. Participants of the Cardiovascular Risk Factors, Aging, and Dementia (CAIDE) study were derived from random, population-based samples previously studied in 1972, 1977, 1982 or 1987. After an average follow-up of 21 years, 1449 individuals (72.5%) aged 65–79 years were re-examined in 1998. The apoE ε4 allele was an independent risk factor for dementia/AD even after adjustments for sociodemographic, lifestyle and vascular factors (odds ratio [OR] = 2.83, 95% confidence interval [CI] ε1.61–4.97). Physical inactivity, alcohol drinking and smoking increased the risk of dementia/AD particularly among the apoE ε4 carriers. Furthermore, low–moderate intake of polyunsaturated, and moderate–high intake of saturated fats were associated with an increased risk of dementia/AD more pronouncedly among apoE ε4 carriers. Composite effect of the lifestyle factors was particularly seen among the ε4 carriers (OR = 11.42, 95% CI = 1.94–67.07 in the 4th quartile). Physical inactivity, dietary fat intake, alcohol drinking and smoking at midlife are associated with the risk of dementia and AD, especially among the apoE ε4 carriers. The apoE ε4 carriers may be more vulnerable to environmental factors, and thus, lifestyle interventions may greatly modify dementia risk particularly among the genetically susceptible individuals.