Prostate cancer (PCa) is heterogeneous and contains both differentiated and undifferentiated tumor cells, but the relative functional contribution of these two cell populations remains unclear. Here we report distinct molecular, cellular, and tumor-propagating properties of PCa cells that express high (PSA⁺) and low (PSA^−/lo) levels of the differentiation marker PSA. PSA^−/lo PCa cells are quiescent and refractory to stresses including androgen deprivation, exhibit high clonogenic potential, and possess long-term tumor-propagating capacity. They preferentially express stem cell genes and can undergo asymmetric cell division to generate PSA⁺ cells. Importantly, PSA^−/lo PCa cells can initiate robust tumor development and resist androgen ablation in castrated hosts, and they harbor highly tumorigenic castration-resistant PCa cells that can be prospectively enriched using ALDH⁺CD44⁺α2β1⁺ phenotype. In contrast, PSA⁺ PCa cells possess more limited tumor-propagating capacity, undergo symmetric division, and are sensitive to castration. Altogether, our study suggests that PSA^−/lo cells may represent a critical source of castration-resistant PCa cells.