The recurrence rates of benign phyllodes tumors diagnosed through surgery and ultrasound-guided vacuum-assisted excision (US-VAE) were evaluated. A total of 146 benign phyllodes tumors diagnosed by surgery (n = 126) or US-VAE (n = 20) in 144 patients who had further follow-up after surgery or US-VAE were included (median follow-up period, 32.3 months; range, 6.7–142.5 months). Comparisons of recurrence rate, interval to recurrence, patient age, initial tumor size, Breast Imaging Reporting and Data System category, or follow-up interval were performed between the surgery and VAE groups and between groups with and without recurrence. Three cases (2.1%, 3/146) had recurrence and all were in the surgery group (2.4%, 3/126). The surgery group demonstrated larger size than the VAE group (median, 25 vs 16 mm; P < 0.001). The median age of women in the surgery group was older than those in the VAE group (39 vs 33 years, P = 0.509). The median age of women with recurrence (n = 3) was older than those without recurrence (n = 143, 49 vs 38 years, P = 0.023). In conclusion, when benign phyllodes tumor is unexpectedly diagnosed at US-VAE, if there is no residual lesion at US, clinical follow-up rather than further surgery might be recommended.