Language impairment is common in prodromal stages of Alzheimer’s disease (AD) and progresses over time. However, the genetic architecture underlying language performance is poorly understood. To identify novel genetic variants associated with language performance, we analyzed brain MRI and performed a genome-wide association study (GWAS) using a composite measure of language performance from the Alzheimer’s Disease Neuroimaging Initiative (ADNI; n = 1560). The language composite score was associated with brain atrophy on MRI in language and semantic areas. GWAS identified GLI3 (GLI family zinc finger 3) as significantly associated with language performance (p < 5 × 10−8). Enrichment of GWAS association was identified in pathways related to nervous system development and glutamate receptor function and trafficking. Our results, which warrant further investigation in independent and larger cohorts, implicate GLI3, a developmental transcription factor involved in patterning brain structures, as a putative gene associated with language dysfunction in AD.