There is a growing consensus that social cognition and behavior emerge from interactions across distributed regions of the “social brain”. Researchers have traditionally focused their attention on functional response properties of these gray matter networks and neglected the vital role of white matter connections in establishing such networks and their functions. In this article, we conduct a comprehensive review of prior research on structural connectivity in social neuroscience and highlight the importance of this literature in clarifying brain mechanisms of social cognition. We pay particular attention to three key social processes: face processing, embodied cognition, and theory of mind, and their respective underlying neural networks. To fully identify and characterize the anatomical architecture of these networks, we further implement probabilistic tractography on a large sample of diffusion-weighted imaging data. The combination of an in-depth literature review and the empirical investigation gives us an unprecedented, well-defined landscape of white matter pathways underlying major social brain networks. Finally, we discuss current problems in the field, outline suggestions for best practice in diffusion-imaging data collection and analysis, and offer new directions for future research.