Apolipoprotein E (APOE) ε4 has been associated with smaller hippocampal volumes in healthy aging, while findings in young adults are inconclusive. Previous studies have mostly used univariate methods, and without considering potential anterior/posterior differences. Here, we used a multivariate method, partial least squares, and assessed whole-brain structural covariance of the anterior (aHC) and posterior (pHC) hippocampus in young adults (n = 97) as a function of APOE ε4 status and sex. Two significant patterns emerged: (1) specific structural covariance of the aHC with frontal regions, temporal and occipital areas in APOE ε4 women, whereas the volume of both the aHC and pHC in all other groups co-varied with frontal, parietal and cerebellar areas; and (2) opposite structural covariance of the pHC in ε4 carriers compared to the aHC in non-carriers, with the pHC of ε4 carriers covarying with parietal and frontal areas, and the aHC of ε4 non-carriers covarying with motor areas and the middle frontal gyrus. APOE ε4 has in young adults been associated with better episodic and spatial memory, functions involving the aHC and pHC, respectively. We found no associations between structural covariance and performance, suggesting that other factors underlie the performance differences seen between carriers and non-carriers. Our findings indicate that APOE ε4 carriers and non-carriers differ in hippocampal organization and that there are differences as a function of sex and hippocampal segment. They stress the need to consider the hippocampus as a heterogeneous structure, and highlight the benefits of multivariate methods in assessing group differences in the brain.