We examined the prognostic role of PD-1+ and CD8+ tumor infiltrating lymphocytes (TILs), and PD-L1+ cells in patients with squamous cell carcinoma of the head and neck (SCCHN) treated with surgery and postoperative chemoradiotherapy (CRT). FFPE samples from 161 patients were immunohistochemically stained for PD-1, CD8 and PD-L1. The immune marker expression was correlated with clinicopathologic characteristics, and overall survival (OS), local progression-free survival (LPFS) and distant metastases free-survival (DMFS), also in the context of HPV16 DNA/p16 status. The median follow-up was 48 months (range: 4–100). The 2-year-OS was 84.1% for the entire cohort. High PD-1 and PD-L1 expression were more common in patients with positive HPV16 DNA (p < 0.001 andp = 0.008, respectively) and high infiltration by CD8+ TILs (p < 0.001 for both markers). High PD-L1 expression correlated with superior OS (p = 0.025), LPFS (p = 0.047) and DMFS (p = 0.048) in multivariable analysis, whereas no significance could be demonstrated for PD-1. Patients with CD8high/PD-L1high expression had favorable outcome (p < 0.001 for all endpoints) compared to other groups. We validated the superior OS data on CD8high/PD-L1high using the Cancer Genome Atlas TCGA dataset (n = 518;p = 0.032). High PD-L1 expression was a favorable prognostic marker in HPV16-negative but not HPV16-positive patients. In conclusion, HPV-positive tumors showed higher expression of immune markers. PD-L1 expression constitutes an independent prognostic marker in SCCHN patients post-adjuvant CRT. In conjunction with CD8 status, these data provide an important insight on the immune contexture of SCCHN and are directly relevant for future treatment stratification with PD-1/PD-L1 immune checkpoint inhibitors to complement CRT.