Differential brain structural abnormalities between bipolar disorder (BD) and major depressive disorder (MDD) may reflect different pathological mechanisms underlying these two brain disorders. However, few studies have directly compared the brain structural properties, especially in white matter (WM) tracts, between BD and MDD. Using automated fiber-tract quantification (AFQ), we utilized diffusion tensor images (DTI) from 67 unmedicated depressed patients, including 31 BD and 36 MDD, and 45 healthy controls (HC) to create fractional anisotropy (FA) tract profiles along 20 major WM tracts. Then, we compared between-group differences in FA values at each node along the fiber tracts. To differentiate the BD and the MDD, we enrolled the diffusion measures of the tract profiles into support vector machine (SVM), a type of machine learning algorithm. The BD showed lower FA in the insular cortex portion of the right uncinate fasciculus (UF) compared to the MDD and in the prefrontal lobe portion of the right UF compared to the HC. The MDD showed lower FA in the prefrontal lobe portion of the left anterior thalamic radiation (ATR) compared to the HC. Using the SVM approach, we found the FA tract profile of the left ATR can be used to differentiate the BD and the MDD at an accuracy up to 68.33% (p = 0.018). These findings suggested that the BD and the MDD may be characterized by different abnormalities in specific segments of brain WM tracts, especially in two frontal-situated tracts, the right UF and the left ATR.