Dietary modifications such as caloric restriction (CR) have been suggested as a means to improve memory and prevent age-related decline. However, it is unclear whether those effects remain stable over time or are related specifically to negative energy balance during the weight loss phase of CR. Using a randomized interventional design, we investigated changes in recognition memory and neural correlates in postmenopausal obese women (n = 19): 1) after intense weight loss in the course of a 12-week low-caloric diet (reduced body weight and negative energy balance) and 2) after having sustained the reduced weight over 4 more weeks (reduced body weight, but energy balance equilibrium). Participants were contrasted to a control group (n = 18) instructed not to change dietary habits. In the CR group, we found improved recognition memory, paralleled by increased gray matter volume in inferior frontal gyrus and hippocampus, and augmented hippocampal resting-state functional connectivity to parietal areas. Moreover, effects were specific for transient negative energy balance and could not be detected after subsequent weight maintenance. Our data demonstrate for the first time in humans that beneficial effects of CR on brain structure and function are due to weight loss rather than an overall reduced weight.