The hippocampus is a structure that is critical for memory. Previous studies have shown that age-related differences in specialization along the longitudinal axis of this structure (i.e., subregions) and within its internal circuitry (i.e., subfields) relate to age-related improvements in memory in school-age children and adults. However, the influence of age on hippocampal development and its relations with memory ability earlier in life remains under-investigated. This study examined effects of age and sex on hippocampal subregion (i.e., head, body, tail) and subfield (i.e., subiculum, CA1, CA2–4/DG) volumes, and their relations with memory, using a large sample of 4- to 8-year-old children. Results examining hippocampal subregions suggest influences of both age and sex on the hippocampal head during early childhood. Results examining subfields within hippocampal head suggest these age effects may arise from CA1, whereas sex differences may arise from subiculum and CA2–4/DG. Memory ability was not associated with hippocampal subregion volume but was associated with subfield volume. Specifically, within the hippocampal head, relations between memory and CA1 were moderated by age; in younger children bigger was better, whereas in older children smaller was superior. Within the hippocampal body, smaller CA1 and larger CA2–4/DG contributed to better memory performance across all ages. Together, these results shed light on hippocampal development during early childhood and support claims that the prolonged developmental trajectory of the hippocampus contributes to memory development early in life.