The Apolipoprotein E4 (ApoE-ε4) allele has been suggested as the main risk factor for late onset Alzheimer's disease (AD), whereas the ApoE-ε2 allele has been proposed as a protective factor. These proposals have increased the interest in the effect of the ApoE genotype in healthy people. Additionally, high cortisol levels have been related to negative effects on cognition. However, few studies have investigated the relationship between cognitive performance and cortisol, taking into account the different ApoE alleles. For this reason, the aim of this study was to evaluate different cognitive domains (declarative and working memory, attention, and executive function) and their relationship with cortisol, considering the ApoE-ε2, ApoE-ε3, and ApoE-ε4 alleles in healthy older people (55–77 years old). Two saliva samples were collected during the neuropsychological session to obtain cortisol levels and the ApoE genotype. Results showed an association between the ApoE genotype and declarative memory, specifically learning ability, where ApoE-ε2 group performed better than ApoE-ε4 and ApoE-ε3 groups. No differences in cortisol levels were obtained considering the ApoE genotype. In addition, higher mean cortisol levels were related to a worse performance on declarative memory, for the whole sample, and when considering the three allelic variation, for the ApoE-ε4 group. On the contrary, an increase of cortisol levels during the neuropsychological session was associated to a better performance on declarative memory for the whole sample, and for the ApoE-ε3 group when considering the three alleles. Besides, ApoE-ε3 group also showed an association between higher mean cortisol levels and a better attention performance. Therefore, our results suggest that carrying the ApoE-ε4 allele may be a vulnerability factor in the adverse effects of Hypothalamic-Pituitary-Adrenal (HPA) axis dysregulation on cognition during aging, while ApoE-ε3 allele could be associated to a more adaptive HPA axis response.