Although PD-1/PD-L1 immunotherapy has been used successfully in treating many cancers, metastatic colorectal cancer (CRC) patients are not as responsive. B7-H3 is a promising target for immunotherapy and we found it to have the highest expression among B7-CD28 family members in CRC. Thus, the aim of the present study was to investigate B7-H3 expression in a large CRC cohort. B7-H3, B7-H4, and PD-L1 protein levels and differential lymphocyte infiltration were evaluated in tissue microarrays from 805 primary tumors and matched metastases. The relationships between immune markers, patient characteristics, and survival outcomes were determined. B7-H3 (50.9%) was detected in more primary tumors than B7-H4 (29.1%) or PD-L1 (29.2%), and elevated B7-H3 expression was associated with advanced overall stage. Co-expression of B7-H3 only with B7-H4 or PD-L1 was infrequent in primary tumors (6.3%, 5.7%, respectively). Moreover, B7-H3 in primary tumors was positively correlated with their respective expression at metastatic sites (ρ = 0.631; p < 0.001). No significant relationships between B7-H4 and PD-L1 and survival were observed; however, B7-H3 overexpression in primary tumors was significantly related to decreased disease-free survival. A positive relationship between B7-H3 expression and high density CD45RO T cell was observed in primary tumors, whereas B7-H4 and PD-L1 overexpression were related to CD3 T-cell infiltration. In conclusion, compared with B7-H4 and PD-L1, B7-H3 expression exhibited a higher prevalence and was significantly related to aggressiveness, worse prognosis and CD45RO T-cell infiltration in primary tumors. Further exploration of this potential target of immunotherapy in CRC patients is warranted.