To determine whether tumor microenvironments affect the clinical response to anti-PD-1 therapy in non-small cell lung cancer, we investigated the expression level of PD-L1 and tumor infiltrating lymphocytes and elucidate their predictive role. Thirty-eight pretreatment and two post-treatment specimens from 36 advanced, treatment-refractory non-small cell lung cancer patients who underwent PD-1 blockade therapy were analyzed. PD-L1 expression by tumor cells and the distribution of CD3, CD8, CD4, FOXP3 and PD-1 positive tumor infiltrating lymphocytes were immunohistochemically assessed and counted using digital image analyzer. CD3+ and CD8+ T cells were distributed more in PD-L1 positive group compared to PD-L1 negative group. Conversely, EGFR mutant group showed fewer CD3+ T cells than EGFR-naïve group. The patients in the clinical benefit group with PD-1 blockade showed a higher number of CD3+, CD8+ T cells and a higher CD8+/CD3+ T cell ratio (p = 0.003, p = 0.001, and p = 0.042) and a lower FOXP3+/CD8+ T cell ratio compared to non-responders (p = 0.001). In multivariate logistic regression analysis, increased CD3+ T cell infiltration and low FOXP3+/CD8+ T cell ratio were found to be independent predictors of clinical benefit with PD-1 blockade (p = 0.014 and p = 0.03, respectively). Using receiver operating characteristic curves, levels of CD3+ T cells and FOXP3+/CD8+ T cell ratio that provide the best distinguishing point between responder versus non-responder to PD-1 blockade were 617.5/mm and 25%, respectively (p = 0.007 and p = 0.003). Considering that 1 mm is about 5 high power fields (HPF), a good response to the PD-1 blockade can be expected when the number of CD3 T cells is observed to be 120 per HPF and when CD8+ T cells and FOXP3+ T cells are present at a ratio greater than 4:1. Tumor infiltrating lymphocytes might become a promising biomarker as an independent predictive factor of response to PD-1 blockade that may also guide therapeutic decisions.