Asian countries are currently witnessing unprecedented increase in physical inactivity and subsequent negative health outcomes; however, few cross-country studies documenting this trend exist. This paper presents the findings of a nationally representative sample, based on the East Asian Social Survey in 2011. The study sought to examine the association of social identity factors, such as objective socio-economic position, perceived social status and neighborhood quality with physical inactivity, while controlling for psychosocial and physical health. A sample of 5222 adults living in urban areas across China, Taiwan, and South Korea were surveyed. Methods: Multivariate nested logistic regressions were constructed. Results: Perceived social status was positively associated with physical activity. Gender difference in physical activity was significant, and this difference widened as educational levels increased. Class division in physical activity was also found. Perceived physical and social features of neighborhood such as suitability for walking and jogging, air quality, and help from neighbors were to different degrees associated with physical inactivity. Conclusion: Gender, marital status, education and perceived social status were common factors associated with physical inactivity in East Asian countries. Perceived urban neighborhood quality is particularly important for Chinese people to stay physically active. Cultural-behavioral norms for physical activity associated with gender and social status call for more studies on cultural perspective for health behaviors in cross-cultural contexts.