Screening for colorectal cancer (CRC) is associated with reduced CRC mortality, but low screening rates have been reported in several settings. The aim of the study was to assess predictors of low CRC screening in Switzerland. A retrospective cohort of a random sample of 940 patients aged 50–80 years followed for 2 years from four Swiss University primary care settings was used. Patients with illegal residency status and a history of CRC or colorectal polyps were excluded. We abstracted sociodemographic data of patients and physicians, patient health status, and indicators derived from RAND’s Quality Assessment Tools from medical charts. We defined CRC screening as colonoscopy in the last 10 years, flexible sigmoidoscopy in the last 5 years, or fecal occult blood testing in the last 2 years. We used bivariate and multivariate logistic regression analyses. Of 940 patients (mean age 63.9 years, 42.7% women), 316 (33.6%) had undergone CRC screening. In multivariate analysis, birthplace in a country outside of Western Europe and North America [odds ratio (OR) 0.65, 95% confidence interval (CI) 0.45–0.97], male sex of the physician in charge (OR 0.67, 95% CI 0.50–0.91), BMI 25.0–29.9 kg/m2 (OR 0.66, CI 0.46–0.96) and at least 30.0 kg/m2 (OR 0.61, CI 0.40–0.90) were associated with lower CRC screening rates. Obesity, overweight, birthplace outside of Western Europe and North America, and male sex of the physician in charge were associated with lower CRC screening rates in Swiss University primary care settings. Physician perception of obesity and its impact on their recommendation for CRC screening might be a target for further research.