Individuals carrying the *E4 allele of the apolipoprotein E gene (APOE) are at increased risk of developing Alzheimer's disease (AD). However, the biological mechanisms underlying this association are still unclear because of the complexity of the pathological processes that cause AD. Furthermore, the effect of APOE genotype on development, maintenance and aging of the normal brain is poorly understood because of the strong bias toward studying disease associations. In vivo techniques such as neuroimaging and cognitive testing offer valuable insights into the effects of APOE genotype on brain structure and function in healthy and clinical populations. We review the evidence from in vivo studies that APOE *E4, in addition to increasing the chance of age-related pathological events, is associated with age-independent non-pathological changes in brain physiology, some of which make the brain less resilient to neurodegenerative processes. We argue that the interaction between the APOE-dependent non-pathological vulnerabilities and age-related pathological changes is one mechanism that can trigger neurodegeneration, resulting in AD and other complex phenotypes.