Multiple studies have described associations between infiltrating immune cells and prognosis in cancer; however, the clinical relevance has most often been attributed to the T-cell linage. This study aimed to further investigate the clinicopathological correlates and prognostic impact of B cell and plasma cell infiltration in CRC. Immunohistochemical expression of CD20, CD138 and immunoglobulin kappa C (IGKC) was analysed in tissue microarrays with tumours from 557 incident cases of CRC from a prospective population-based cohort. Kaplan–Meier analysis and Cox regression analysis were used to determine the impact of CD20, CD138 and IGKC expression on 5-year overall survival. Immune cell-specific CD20, CD138, and IGKC expression correlated significantly with lower T-stage (p< 0.001,p< 0.001, andp= 0.006, respectively). A higher density of CD20+ cells correlated significantly with an improved OS (HR = 0.53, 95% CI 0.36–0.78), remaining significant in multivariable analysis adjusted for age, TNM stage, differentiation grade and vascular invasion (HR = 0.51; 95% CI 0.33–0.80). Immune cell-specific CD138 and IGKC expression correlated significantly with an improved OS in univariable Cox regression analysis; however, these associations did not remain significant in multivariable analysis. Finally, tumour cell-specific CD138 expression was found to be an independent factor of poor prognosis (HR 1.52; 95% CI 1.03–2.24). The results from the present study demonstrate that B cell infiltration in CRC has a significant impact on tumour progression and prognosis. These findings supplement and extend the current knowledge of the immune landscape in colorectal cancer, and merit further study.