The presence of tumor‐infiltrating lymphocytes (TILs) is associated with favorable long‐term outcome in breast cancer. However, little is known about changes in TILs during metastatic progression. To confirm our hypothesis that malignant tumors escape from the host immune system during metastasis, we evaluated the percentage of TILs in paired samples of primary and metastatic breast tumors. We retrospectively identified 25 patients with human epidermal growth factor receptor‐2 (HER2⁺, n = 14) and triple negative (TN, n = 11) early breast cancer diagnosed between 1990 and 2009 at Tokai University Hospital (Isehara, Japan) and who subsequently experienced regional or distant recurrence confirmed by tumor biopsy/resection. Hematoxylin–eosin‐stained slides of these paired samples were evaluated for stromal TILs. Immunohistochemical staining was carried out using primary antibodies against CD4, CD8, Foxp3, programmed cell death ligand 1 (PD‐L1), PD‐L2, and HLA class I for characterizing the TILs and breast tumors. The percentage of TILs in the primary tumors was significantly higher (average 34.6%) than that in metastatic tumors (average 15.7%) (paired t‐test, P = 0.004) and that of CD8⁺ and CD4⁺ T cells significantly decreased from primary to metastatic tumors (paired t‐test, P = 0.008 and P = 0.026, respectively). The PD‐L1, PD‐L2, and HLA class I antibody expression changed from positive to negative and vice versa from the primary to the metastatic tumors. Tumors at first metastatic recurrence in HER2⁺ and TN breast cancers have a lower percentage of TILs and CD8⁺ and CD4⁺ T cells compared to primary tumors, which indicates that immune escape plays a role in tumor progression.