Phyllodes tumor or cystosarcoma phyllodes is a rare fibroepithelial neoplasm which arises from the periductal stroma of the breast. They are classified as benign, borderline, and malignant based on the histologic features. However, all phyllodes tumor (PT) subtypes are regarded as having malignant potential and correct diagnosis is important for surgical management and optimal care. This study is a retrospective review of 76 women diagnosed as PT with highlights on the imaging characteristics, pathology, and surgical treatment over a 7-year period in a tertiary medical center of urban population in Malaysia. There were 45 benign, 16 borderline, and 15 malignant PT. The median age for benign PT was 43, borderline 48.5, and malignant 42 years. The Malay ethnic group constitute 52.6% of cases, with 27.6% and 18.4% in Chinese and Indian ethnic groups, respectively. On mammograms, most benign (64.3%) and 33.3% of malignant PT showed high-density lesions. Calcifications were only seen in 2 benign PT. On ultrasound, 86% of benign PT was well-circumscribed whilst 50.0% of malignant PT had irregular outline. Cystic spaces were seen in 40.0% of malignant and 9.5% of benign PT. 80% of malignant PT lesions were heterogenous. Malignant PT demonstrates tumor heterogeneity, cystic spaces, and posterior acoustic enhancement on ultrasound. Half of malignant PT showed regular borders on ultrasound and appear well circumscribed on mammogram. A total of 46 patients had wide local excision or excision biopsy whilst 30 underwent mastectomy as primary treatment. The majority of the borderline and malignant PTs in our study (75.0% and 85.7% respectively) and only 5 out of the 43 (11.6%) benign PT underwent mastectomy. There were 2 tumor recurrence in the benign PT group and 1 case in the borderline and malignant group respectively.