Stress is a potent modulator of learning and memory. Factors contributing to whether stress aids or impairs memory are timing of the stressor, memory stage, form of memory studied, and sex of the subjects. The female sex hormone 17-beta-estradiol (E2) has widespread effects in the brain and affects hippocampus-dependent memory in animals. In humans, the interaction between stress effects and E2 has not been widely studied. We report data from a healthy sample divided into 3 hormone-status groups: free-cycling women in the early follicular phase (EF: low E2, low progesterone [P4]), or during midcycle (MC: high E2, low P4), and men. Participants within each hormone-status group were randomly assigned to a psychosocial stressor or a control treatment 37 min before encoding a short story of neutral content. We found a Hormone status × Stress × Time (immediate, 35-min, 24-h delayed recall) interaction. Irrespective of time, hormone status mattered only after stress treatment: stressed early follicular women had poorer recall compared to stressed men and midcycle women. Only in the early follicular group, recall was negatively correlated with increases in salivary cortisol, but not with blood levels of E2 and P4. To uncover changes beyond immediate recall, we computed the individual percent change relative to immediate recall and repeated the analysis for these adjusted 35-min and 24-h data. Despite the lack of a stress effect in raw data, memory in stressed men was more stable over time (35-min and 24-h delay) than in unstressed men. In contrast, stressed EF-women (vs. control) recalled less at the 35-min and (as a trend) at the 24-h delay. Stressed MC-women (vs. control) showed less recall only at the 35-min delay while compensating this stress effect after a 24-h consolidation interval. Overall, results suggest that women in high-E2 midcycle phase could be less vulnerable to effects of pre-learning stress on declarative memory encoding and consolidation.