Automated 3D breast ultrasound (ABUS) is a new popular modality as an adjunct to mammography for detecting cancers in women with dense breasts. In this paper, a multi-stage computer aided detection system is proposed to detect cancers in ABUS images. In the first step, an efficient despeckling method called OBNLM is applied on the images to reduce speckle noise. Afterwards, a new algorithm based on isocontours is applied to detect initial candidates as the boundary of masses is hypo echoic. To reduce false generated isocontours, features such as hypoechoicity, roundness, area and contour strength are used. Consequently, the resulted candidates are further processed by a cascade classifier whose base classifiers are Random Under-Sampling Boosting (RUSBoost) that are introduced to deal with imbalanced datasets. Each base classifier is trained on a group of features like Gabor, LBP, GLCM and other features. Performance of the proposed system was evaluated using 104 volumes from 74 patients, including 112 malignant lesions. According to Free Response Operating Characteristic (FROC) analysis, the proposed system achieved the region-based sensitivity and case-based sensitivity of 68% and 76% at one false positive per image.