A consensus on Bantu-speaking populations being genetically similar has emerged in the last few years, but the demographic scenarios associated with their dispersal are still a matter of debate. The frontier model proposed by archeologists postulates different degrees of interaction among incoming agropastoralist and resident foraging groups in the presence of “static” and “moving” frontiers. By combining mitochondrial DNA and Y chromosome data collected from several southern African populations, we show that Bantu-speaking populations from regions characterized by a moving frontier developing after a long-term static frontier have larger hunter-gatherer contributions than groups from areas where a static frontier was not followed by further spatial expansion. Differences in the female and male components suggest that the process of assimilation of the long-term resident groups into agropastoralist societies was gender biased. Our results show that the diffusion of Bantu languages and culture in Southern Africa was a process more complex than previously described and suggest that the admixture dynamics between farmers and foragers played an important role in shaping the current patterns of genetic diversity.