Cancer evolution is predominantly studied by focusing on differences in the genetic characteristics of malignant cells within tumors. However, the spatiotemporal dynamics of clonal outgrowth that underlie evolutionary trajectories remain largely unresolved. Here, we sought to unravel the clonal dynamics of colorectal cancer (CRC) expansion in space and time by using a color-based clonal tracing method. This method involves lentiviral red-green-blue (RGB) marking of cell populations, which enabled us to track individual cells and their clonal outgrowth during tumor initiation and growth in a xenograft model. We found that clonal expansion largely depends on the location of a clone, as small clones reside in the center and large clones mostly drive tumor growth at the border. These dynamics are recapitulated in a computational model, which confirms that the clone position within a tumor rather than cell-intrinsic features, is crucial for clonal outgrowth. We also found that no significant clonal loss occurs during tumor growth and clonal dispersal is limited in most models. Our results imply that, in addition to molecular features of clones such as (epi-)genetic differences between cells, clone location and the geometry of tumor growth are crucial for clonal expansion. Our findings suggest that either microenvironmental signals on the tumor border or differences in physical properties within the tumor, are major contributors to explain heterogeneous clonal expansion. Thus, this study provides further insights into the dynamics of solid tumor growth and progression, as well as the origins of tumor cell heterogeneity in a relevant model system.