The aim of the present study was to define the prognostic role of baseline serum albumin (BSA) in colorectal cancer (CRC) across tumor–node–metastasis (TNM) stages and other well defined prognostic factors. Many prognostic models in medicine employ BSA to define or refine treatments in very specific settings; in CRC, BSA has been found to be a prognostic factor as well. A retrospective cohort study of consecutive patients with CRC demonstrated by biopsy, who attended a cancer center during a 7-year period. Multivariate analysis was utilized to define prognostic factors associated with overall survival (OS) employing the Cox model. In this retrospective cohort study, 1465 patients were included; 46.6% were females and 53.4% males (mean age, 59.1 years). Mean BSA was inversely correlated with TNM stages. By multivariate analysis, it was an independent explanatory variable. TNM stages, “R” classification, age, lymphocyte count, neutrophil/platelet ratio, neutrophil/lymphocyte ratio, platelet/lymphocyte ratio, postoperative morbidity, and BSA were independently associated with OS. Morbidities, surgery type, chemotherapy, and radiotherapy were considered confounders after adjusting by TNM stages. BSA is a significant and independent prognostic factor in patients with CRC, and its effect is maintained across TNM strata and other well known clinical prognostic factors. It can be easily used in prognostic models and should be employed to stratify prognosis in therapeutic randomized clinical trials.