Acoustic radiation force impulse (ARFI) is a newly developed elastography technique that uses acoustic radiation force to provide additional stiffness information to conventional sonography. A computer-aided diagnosis (CAD) system was proposed to automatically specify the tumor boundaries in ARFI images and quantify the statistical stiffness information to reduce user dependence. The level-set segmentation was used to delineate tumor boundaries in B-mode images, and the segmented boundaries were then mapped to the corresponding area in ARFI images for a gray-scale calculation. A total of 61 benign and 51 malignant tumors were evaluated in the experiment. The CAD system based on the proposed ARFI features achieved an accuracy of 80% (90/112), a sensitivity of 80% (41/51), and a specificity of 80% (49/61), which is significantly better than that of the quantitative B-mode features (p < 0.05). The ARFI features were further combined with the B-mode features, including shape and texture features, to further improve performance (area under the curve [AUC], 0.90 vs. 0.86). In conclusion, the CAD system based on the proposed ARFI features is a promising and efficient diagnostic method.