Endoscopic mucosal biopsies of the ampulla of Vater (AmpBx) are obtained to histologically assess for dysplasia or carcinoma. However, biopsy material is often scant and a host of factors can induce histologic changes that pose diagnostic challenges. We sought to investigate observer variability in interpretation of AmpBx and the impact clinical data may have on diagnostic interpretation. Thirty-one cases from institutional archives were selected, including 12 cases of reactive atypia (RA), 8 indefinite for dysplasia (ID), and 11 showing low-grade dysplasia (LGD). Slides were independently reviewed at 3 time points with and without clinical information by 6 pathologists who categorized the biopsies RA, ID, or LGD. Following the reviews, intraobserver and interobserver agreement was assessed. Review of AmpBx without clinical data showed fair (κ, 0.27), poor (κ, 0.07), and good (κ, 0.42) interobserver agreement for diagnoses of RA, ID, and LGD, respectively. Interobserver agreement improved for LGD (κ, 0.66 and 0.73) when clinical information was provided; however, agreement remained fair for RA (κ, 0.4 and 0.42) and poor-to-fair for ID (κ, 0.17 and 0.25). When follow-up data were reviewed, all cases that reached unanimous agreement had that diagnosis substantiated by subsequent endoscopic or histologic findings. The same was true of 13 of 19 cases that reached majority consensus. Given the potential clinical consequences of these diagnoses combined with the significant intraobserver and interobserver variability found in this study, we conclude that better-defined diagnostic criteria and consensus reads on difficult cases would assist in the histologic assessment of these challenging cases.