Y-chromosome-specific haplotypes (Y-haplotypes) constructed using single nucleotide polymorphisms (Y-SNPs) in the MSY (male-specific region of the Y-chromosome) are valuable in population genetic studies. But sequence variants in the yak MSY region have been poorly characterized so far. In this study, we screened a total of 16 Y-chromosome-specific gene segments from the ZFY, SRY, UTY, USP9Y, AMELY and OFD1Y genes to identify Y-SNPs in domestic yaks. Six novel Y-SNPs distributed in the USP9Y (g.223C>T), UTY19 (g.158A>C and g.169C>T), AMELY2 (g.261C>T), OFD1Y9 (g.165A>G) and SRY4 (g.104G>A) loci, which can define three Y-haplotypes (YH1, YH2 and YH3) in yaks, were discovered. YH1 was the dominant and presumably most ancient haplotype based on the comparison of UTY19 locus with other bovid species. Interestingly, we found informative UTY19 markers (g.158A>C and g.169C>T) that can effectively distinguish the three yak Y-haplotypes. The nucleotide diversity was 1.7 × 10−4 ± 0.3 × 10−4, indicating rich Y-chromosome diversity in yaks. We identified two highly divergent lineages (YH1 and YH2 vs. YH3) that share similar frequencies (YH1 + YH2: 0.82–0.89, YH3: 0.11–0.18) among all three populations. In agreement with previous mtDNA studies, we supported the hypothesis that the two highly divergent lineages (YH1 and YH2 vs. YH3) derived from a single gene pool, which can be explained by the reunion of at least two paternal populations with the divergent lineages already accumulated before domestication. We estimated a divergence time of 408 110 years between the two divergent lineages, which is consistent with the data from mitochondrial DNA in yaks.