Background: Smoking and physical inactivity are linked to mortality, but it is not known whether the association between smoking and mortality is affected by the amount and intensity of physical activity. We examined the joint associations of smoking and physical activity with mortality, while taking key covariates into account. Methods: We linked survey data, collected in 2000–2002 from among 40–60-year-old employees of the City of Helsinki, Finland, with complete register data on all-cause mortality from Statistics Finland (n = 6390, 79% women, response rate 67%). Smoking, leisure-time physical activity and covariates (sociodemographic factors, problem drinking, body mass index and self-rated health) were measured at baseline. We fitted Cox regression models (hazard ratios, HR, 95% confidence intervals, CI), and the follow-up continued until the end of 2013. No gender interactions were found. Results: A total of 228 deaths occurred during the follow-up. Smokers were at an increased risk of mortality after full adjustments, but the risk was higher among inactive (HR 3.27, 95% CI 2.05–5.22) and moderately active smokers (HR 2.37, 95% CI 1.49–3.79) than among vigorously active non-smokers. The excess risk for vigorously active smokers, or for inactive or moderately active non-smokers, could not be confirmed. Conclusion: The highest mortality risk was found among physically inactive or moderately active smokers. Prevention of smoking and engaging in vigorous physical inactivity among smokers might prevent mortality during working age.