Neutrophil to lymphocyte ratio (NLR) has been found to be significantly associated with pancreatic cancer (PC) survival. However, no existing studies discussed the association between neutrophil count, lymphocyte count, and PC survival jointly. In this study, we aimed to analyze the influence of neutrophil and lymphocyte counts measured at disease diagnosis on the overall survival (OS) of PC. A total of 288 PC patients diagnosed between January 1, 2012, and December 31, 2013, were retrospectively selected from a population-based electronic inpatients database. Multivariate Cox model and restricted cubic spline (RCS) were used to estimate the associations between neutrophil count, lymphocyte count, and OS of PC. We found that a decreased lymphocyte count at diagnosis was significantly associated with OS of PC: for PC patients whose lymphocyte counts were less than 1.5 × 109/L, the hazard ratio (HR) was 1.82 (95% confidence interval: 1.37–2.40). Although abnormally increased baseline neutrophil count in general was not associated with OS of PC, RCS found a prominently deteriorated survival for PC patients whose baseline neutrophil counts were close to the cutoff point (7.0 × 109/L). Our study results indicate that neutrophil and lymphocyte counts at diagnosis may have prognostic relevance in PC survival, especially lymphocyte count. The clinical significance of neutrophil inhibition and lymphocyte promotion treatments in PC patients should be further discussed.