Apolipoprotein E (APOE) ε4 allele is the genetic risk factor with the most established evidence for sporadic Alzheimer’s disease. Previous neuroimaging studies have demonstrated insufficiently consistent functional and structural changes among healthy APOE ε4 carriers when compared to non-carriers. Here, in a cognitively intact elderly group (a total of 110: 45 APOE ε4 carriers, 65 non-carriers), we aimed to investigate the potential role of APOE ε4 in the modulation of grey matter activity, white matter integrity, and brain morphology before the development of clinically significant symptoms and signs, by methods of: amplitude of low frequency fluctuations and regional homogeneity analysis based on resting state fMRI, and fiber tractography approach based on diffusion tensor imaging. Our results revealed that compared to non-carriers, APOE ε4 carriers showed: (1) an inconsistent pattern of activity change in the default mode network, including increased gray matter activity in anterior cingulate cortex and medial prefrontal cortex and decreased activity in precuneus; (2) lower mean diffusivity (MD) in fibers of corona radiata and corpus callosum, and lower axial diffusivity in genu of corpus callosum; and (3) significant positive correlation between the MD value of the right superior corona radiate and gross white matter volume; significant negative correlation between the MD value of the right superior corona radiate and Mini-Mental State Examination (MMSE) score. Our results suggested that APOE ε4 gene can modulate gray matter activity and white matter integrity in cognitive and memory related regions, even before any clinical or neuropsychic symtoms or signs of imminent disease.