The pituitary gland is a key structure in the hypothalamic–pituitary–gonadal (HPG) axis — it plays an important role in sexual maturation during puberty. Despite its small size, its volume can be quantified using magnetic resonance imaging (MRI). Here, we study a cohort of 962 typically developing adolescents from the Saguenay Youth Study and estimate pituitary volumes using a newly developed multi-atlas segmentation method known as the MAGeT Brain algorithm. We found that age and puberty stage (controlled for age) each predicts adjusted pituitary volumes (controlled for total brain volume) in both males and females. Controlling for the effects of age and puberty stage, total testosterone and estradiol levels also predict adjusted pituitary volumes in males and pre-menarche females, respectively. These findings demonstrate that the pituitary gland grows during adolescence, and its volume relates to circulating plasma-levels of sex steroids in both males and females.