Diffusion tensor imaging studies have provided evidence for white matter (WM) alterations in cocaine users. While polysubstance use is a widespread phenomenon among cocaine users, its role in WM alterations in cocaine users is currently unknown. This study examined the relation between the number of substances that are used(cocaine, alcohol and marijuana) and WM alterations in 67 male non-drug users and 67 male regular cocaine users, who were classified into five groups based on the number of used substances. Diffusion-weighted images were acquired on a 3.0 T magnetic resonance imaging scanner. Using tract-based spatial statistics we demonstrated that there was a negative relation between the number of used substances and fractional anisotropy, a global measure of WM integrity. Also, we demonstrated a positive relation between the number of used substance and radial diffusivity within the prefrontal lobe, suggesting an increase in demyelination with the number of used substances. We did not find a dose-effect between the level of substance use and WM alterations. The results of the current study may reflect the presence of a pre-existing vulnerability to polysubstance use resulting from prefrontal WM abnormalities and related impaired cognitive control although WM alterations because of polysubstance use cannot be fully excluded. This study is an important first step in understanding the problems related to polysubstance use among cocaine users.