The apolipoprotein E (APOE) ε4 allele is the best characterized genetic risk factor for Alzheimer's disease to date. Older APOE ε4 carriers (aged 60 + years) are known to have disrupted structural and functional connectivity, but less is known about APOE-associated network integrity in middle age. The goal of this study was to characterize APOE-related differences in network topology in middle age, as disentangling the early effects of healthy versus pathological aging may aid early detection of Alzheimer's disease and inform treatments. We performed resting state functional magnetic resonance imaging (rs-fMRI) and diffusion tensor imaging (DTI) in healthy, cognitively normal, middle-aged adults (age 40–60; N = 76, 38 APOE ε4 carriers). Graph theoretical analysis was used to calculate local and global efficiency of 1) a whole brain rs-fMRI network; 2) a whole brain DTI network; and 3) the resting state structural connectome (rsSC), an integrated functional-structural network derived using functional-by-structural hierarchical (FSH) mapping. Our results indicated no APOE ε4-associated differences in network topology of the rs-fMRI or DTI networks alone. However, ε4 carriers had significantly lower global and local efficiency of the integrated rsSC compared to non-carriers. Furthermore, ε4 carriers were less resilient to targeted node failure of the rsSC, which mimics the neuropathological process of Alzheimer's disease. Collectively, these findings suggest that integrating multiple neuroimaging modalities and employing graph theoretical analysis may reveal network-level vulnerabilities that may serve as biomarkers of age-related cognitive decline in middle age, decades before the onset of overt cognitive impairment.