We examined how the immune microenvironment molds tumor evolution at different metastatic organs in a longitudinal dataset of colorectal cancer. Through multiplexed analyses, we showed that clonal evolution patterns during metastatic progression depend on the immune contexture at the metastatic site. Genetic evidence of neoantigen depletion was observed in the sites with high Immunoscore and spatial proximity between Ki67⁺ tumor cells and CD3⁺ cells. The immunoedited tumor clones were eliminated and did not recur, while progressing clones were immune privileged, despite the presence of tumor-infiltrating lymphocytes. Characterization of immune-privileged metastases revealed tumor-intrinsic and tumor-extrinsic mechanisms of escape. The lowest recurrence risk was associated with high Immunoscore, occurrence of immunoediting, and low tumor burden. We propose a parallel selection model of metastatic progression, where branched evolution could be traced back to immune-escaping clones. The findings could inform the understanding of cancer dissemination and the development of immunotherapeutics.