The prognostic impact of tumor-infiltrating lymphocytes (TILs) within invasive lobular carcinoma (ILC) remains to be better characterized. In estrogen receptor (ER)-negative invasive ductal carcinomas of no special type (IDC-NST), TILs are associated with good prognosis. The aim of this study was to examine TILs in ILC, with particular focus on prognostic and clinicopathologic features. A cohort comprising 459 consecutive ILCs diagnosed in a single institution from 2005 to 2008 met the eligibility criteria for this study. The percentage of tumor area occupied by TILs was quantified by two breast pathologists and categorized into three groups: no TILs, ≤5%, >5%. Clinicopathologic features were tested by Fisher's exact tests or Chi tests. Overall survival (OS) and invasive disease-free survival (iDFS) were estimated by Kaplan-Meier and Cox proportional hazard statistics. There were 239 TIL-negative cases, 185 cases with ≤5% TILs, and 35 cases with >5% TILs. TILs were associated with younger age, larger tumors, lymph node involvement, poor Nottingham prognostic index, HER2 amplification, multinucleation, and prominent nucleoli (p < 0.05). Poor OS was significantly associated with increasing TILs in the univariate Cox proportional hazards model (p < 0.001) and Kaplan-Meier estimator (p < 0.05, log-rank test). Similar results were observed for iDFS (p = 0.004 for Cox univariate and p = 0.005 for log-rank test). Notably, TILs can identify a subset of ILC patients with poor OS independently of molecular subtype and lymph node metastases (multivariate Cox, p < 0.001, OS hazard ratio (HR) = 4.38 and HR = 6.15, for ≤5% and >5% TILs, respectively, vs. absence of TILs). Prominent nucleoli was the only nuclear feature associated with poor OS (p = 0.05) and iDFS (p = 0.05) in univariate Cox survival analysis. TILs represent a promising new morphologic biomarker associated with poor outcome of ILC, in contrast with that observed in ER-negative IDC-NST.