The identification of cell types of origin for cancer has important implications for tumor stratification and personalized treatment. For prostate cancer, the cell of origin has been intensively studied, but it has remained unclear whether basal or luminal epithelial cells, or both, represent cells of origin under physiological conditions in vivo. Here, we use a novel lineage-tracing strategy to assess the cell of origin in a diverse range of mouse models, including Nkx3.1^+/−; Pten^+/−, Pten^+/−, Hi-Myc, and TRAMP mice, as well as a hormonal carcinogenesis model. Our results show that luminal cells are consistently the observed cell of origin for each model in situ; however, explanted basal cells from these mice can generate tumors in grafts. Consequently, we propose that luminal cells are favored as cells of origin in many contexts, whereas basal cells only give rise to tumors after differentiation into luminal cells.