Disrupted sociability and consequent social withdrawal are (early) symptoms of a wide variety of neuropsychiatric diseases, such as schizophrenia, autism spectrum disorders, depressive disorders and Alzheimer’s disease. The paucity of objective measures to translationally assess social withdrawal characteristics has been an important limitation to study this behavioral phenotype, both in human and rodents. The aim of the present study was to investigate sociability and social withdrawal in rodents using an ethologically valid behavioral paradigm, the Visible Burrow System (VBS). The VBS mimics a natural environment, with male and female rodents housed together in an enclosure where a large open arena is connected to a continuously dark burrow system that includes 4 nest boxes. In this study, mixed-sex colonies of C57BL/6J and of BTBR mice have been investigated (n = 8 mice per colony). Results showed marked differences between the two strains, in terms of sociability as well as social withdrawal behaviors. In particular, BTBR mice performed less social behaviors and have a preference for non-social behaviors compared to C57BL/6J mice. Neurobiologically, the decreased sociability of BTBR was accompanied by reduced GABA and increased glutamate concentrations in brain prefrontal cortex (PFC) and amygdala regions. In conclusion, our study validated the use of the VBS as an ethologically relevant behavioral paradigm in group-housed mice to investigate individual sociability and social withdrawal features and their underlying neurobiology. This paradigm may provide new insights to develop new therapeutic treatments for behavioral dysfunctions that may be relevant across neuropsychiatric diseases.