Clinically relevant histological categorization of fibroepithelial lesions can be a daunting task, especially in a core needle biopsy. Assessment of stromal nuclear atypia, including nuclear pleomorphism and mitotic activity, is a key morphological feature employed to classify fibroepithelial lesions. We describe a case of fibroadenoma with markedly atypical nuclear features in the stromal cells that led to misclassification as phyllodes tumor in the core needle biopsy. Excision showed a fibroadenoma containing pleomorphic stromal giant cells, with occasional mitotic figures, including atypical forms. Aforementioned nuclear findings in a fibroepithelial lesion raise a legitimate question of phyllodes tumor. Knowledge of this pitfall may help avoid overtreatment of an otherwise benign fibroepithelial lesion.