Understanding the mechanisms of immune resistance in pancreatic and ampullary cancers is crucial for the development of suitable biomarkers and effective immunotherapeutics. Our aim was to examine the expression of the immune inhibiting molecules PD-L1, Galectin-9, HVEM, IDO and HLA-G, as well as CD8+ and FoxP3+ tumor infiltrating lymphocytes (TIL), in pancreatic and ampullary cancers, and to relate their individual, as well as their combined expression, to cancer survival. Tumor tissue from 224 patients with resected pancreatic (n = 148) and ampullary (n = 76) cancer was used to construct tissue-microarrays. Expression of immune inhibitory molecules and TIL was examined by immunohistochemistry. We show that immune inhibitory molecules are prevalently expressed. Moreover, high tumor expression of PD-L1 (p = 0.002), Gal-9 (p = 0.003), HVEM (p = 0.001), IDO (p = 0.049), HLA-G (p = 0.004) and high CD8/FoxP3 TIL ratio (p = 0.006) were associated with improved cancer-specific survival. All immune biomarkers, with the exception of IDO, were individually predictive of cancer-specific survival when adjusted for clinicopathologic characteristics. For every additional immune biomarker present survival was almost two-fold prolonged (HR 0.57 95%CI 0.47–0.69,p < 0.0001). When patients with pancreatic and ampullary cancer were analyzed separately the results were similar. We conclude that pancreas and ampullary cancers are rich in expression of immune-inhibitory molecules. These molecules can be targets for future immunotherapeutics, as well as form powerful immunological biomarkers. We propose that such immune biomarker panels be included in future prospective immunotherapy trials.