The level of tumor-infiltrating lymphocytes and presence of tertiary lymphoid structures are significant prognostic and predictive factors in primary breast cancer. However, the understanding about differences in tumor-infiltrating lymphocytes and tertiary lymphoid structures at various metastatic sites or between primary breast tumors and metastatic sites is limited. A total of 335 cases of metastatic breast cancer from four metastatic sites (lung, liver, brain, and ovary) were included. We analyzed the percentages of tumor-infiltrating lymphocytes and presence of tertiary lymphoid structures in the primary and metastatic sites. The mean level of tumor-infiltrating lymphocytes in the lung metastases was higher than in the liver, brain, ovary, and matched primary tumors, while metastatic tumors of the liver and brain showed lower levels of tumor-infiltrating lymphocytes than primary tumors. Tertiary lymphoid structures were only found in the lung and liver, and in cases of brain metastases the change of tertiary lymphoid structures from present to absent significantly affected the level of tumor-infiltrating lymphocytes in metastases compared with that in matched primary tumors. Patients with a lower histological grade, hormone receptor positivity in primary tumors and metastases, a lower level of tumor-infiltrating lymphocytes and absence of tertiary lymphoid structures in primary tumors, a higher level of tumor-infiltrating lymphocytes and presence of tertiary lymphoid structures in metastases, and lung metastases showed significantly better overall survival. Our results showed that metastatic breast tumors in the lung had more tumor-infiltrating lymphocytes than did tumors at other sites and matched primary tumors. In addition, the presence of tertiary lymphoid structures in metastatic sites is a critical factor for the level of tumor-infiltrating lymphocytes.