Masculinization of the altricial rodent brain is driven by estrogen signaling during a perinatal critical period. Genetic deletion of estrogen receptor alpha (Esr1/ERα) results in altered hypothalamic-pituitary-gonadal (HPG) axis signaling and a dramatic reduction of male sexual and territorial behaviors. However, the role of ERα in masculinizing distinct classes of neurons remains unexplored. We deleted ERα in excitatory or inhibitory neurons using either a Vglut2 or Vgat driver and assessed male behaviors. We find that Vglut2-Cre;Esr1lox/lox mutant males lack ERα in the ventrolateral region of the ventromedial hypothalamus (VMHvl) and posterior ventral portion of the medial amygdala (MePV). These mutants recapitulate the increased serum testosterone levels seen with constitutive ERα deletion, but have none of the behavioral deficits. In contrast, Vgat-Cre;Esr1lox/lox males with substantial ERα deletion in inhibitory neurons, including those of the principal nucleus of the bed nucleus of the stria terminalis (BNSTpr), posterior dorsal MeA (MePD), and medial preoptic area (MPOA) have normal testosterone levels, but display alterations in mating and territorial behaviors. These mutants also show dysmasculinized expression of androgen receptor (AR) and estrogen receptor beta (Esr2). Our results demonstrate that ERα masculinizes GABAergic neurons that gate the display of male-typical behaviors.