The pathologic diagnosis is regarded as the final diagnosis of a disease, and pathologic examination based on tumor histology is very important for the accurate assessment of the biological characteristics of tumors. The purpose of this study was to investigate the histologic factors that accurately predict patient outcome among 1042 patients with invasive ductal carcinoma of the breast. Both well-known histologic factors and our proposed histologic factors were examined according to several tumor statuses using multivariate analysis. This study clearly demonstrated that type 4 invasive ductal carcinomas having fibrotic foci and atypical tumor-stromal fibroblasts within the fibrotic foci are significant outcome predictors for lymph node-negative and lymph node-positive, the pathologic UICC-TNM stage II and III, luminal A-subtype, luminal B-subtype, and equivocal HER2 subtype invasive ductal carcinoma patients. Lymph vessel tumor embolus grades 2 and 3 were significant outcome predictors for lymph node-positive, UICC pTNM stages II and III, luminal A-subtype, and triple-negative invasive ductal carcinoma patients (except lymph vessel tumor embolus grade 2 in luminal A-subtype patients). More than 5 mitotic figures in metastatic carcinoma to the lymph nodes was a significant outcome predictor for lymph node-positive, UICC pTNM stage II, and luminal A-subtype invasive ductal carcinoma patients. A fibrotic focus diameter >8 mm was a significant outcome predictor for UICC pTNM stages I and III invasive ductal carcinoma patients. These findings strongly suggest that these histologic factors are very useful for accurately predicting the outcomes of patients with invasive ductal carcinoma of the breast.