As epithelial tumors grow from single cells to a malignant mass of invasive tissue, they must exploit the innate inflammatory response, while evading the adaptive immune system. Prognosis of solid tumors has historically focused on macroscopic features such as size, grade, and mitotic index. It is now clear that prognosis assessment must also consider the stromal and immune cells that surround and infiltrate the tumor. Tumors promote growth, angiogenesis, and tissue remodeling by subverting the normal functions of macrophages and other cells of the innate immune system that inhabit their microenvironment. Simultaneously, tumor cells escape from and inactivate the adaptive immune system by exploiting the mechanisms preventing damaging auto-immune responses in cytotoxic T cells. The presence of CD8+ T cells within epithelial tumors is now a well-supported marker of better prognosis in many tumor types. However, this benefit is counterbalanced by immune regulatory cell populations that promote tumor escape from immune surveillance and metastasis. Therapeutic approaches that kill tumor cells selectively by re-activating immune checkpoints are becoming an established therapeutic option, but it is not yet clear how to identify which patients will benefit from this treatment modality. Evidence is accumulating that identifying the presence of T cell-activating neoantigens, produced by mutated proteins in tumors, will play an important role in checkpoint inhibitor prognosis. This review provides an overview of the evidence that lymphocytic infiltration of tumors has prognostic value in many epithelial tumor types and is linked to the success of chemical and immune checkpoint therapeutic strategies.