Excessive alcohol consumption in humans induces deficits in decision making and emotional processing, which indicates a dysfunction of the prefrontal cortex (PFC). The present study aimed to determine the impact of chronic intermittent ethanol (CIE) inhalation on mouse medial PFC pyramidal neurons. Data were collected 6–8 days into withdrawal from 7 weeks of CIE exposure, a time point when mice exhibit behavioral symptoms of withdrawal. We found that spine maturity in prelimbic (PL) layer 2/3 neurons was increased, while dendritic spines in PL layer 5 neurons or infralimbic (IL) neurons were not affected. Corroborating these morphological observations, CIE enhanced glutamatergic transmission in PL layer 2/3 pyramidal neurons, but not IL layer 2/3 neurons. Contrary to our predictions, these cellular alterations were associated with improved, rather than impaired, performance in reversal learning and strategy switching tasks in the Barnes maze at an earlier stage of chronic ethanol exposure (5–7 days withdrawal from 3 to 4 weeks of CIE), which could result from the anxiety-like behavior associated with ethanol withdrawal. Altogether, this study adds to a growing body of literature indicating that glutamatergic activity in the PFC is upregulated following chronic ethanol exposure, and identifies PL layer 2/3 pyramidal neurons as a sensitive target of synaptic remodeling. It also indicates that the Barnes maze is not suitable to detect deficits in cognitive flexibility in CIE-withdrawn mice.