The intrinsic ability to exhibit self-organizing morphogenetic properties in ex vivo culture may represent a general property of tissue stem cells. Here we show that single luminal stem/progenitor cells can generate prostate organoids in a three-dimensional culture system in the absence of stroma. Organoids generated from CARNs (castration-resistant Nkx3.1-expressing cells) or normal prostate epithelia exhibit tissue architecture containing luminal and basal cells, undergo long-term expansion in culture and exhibit functional androgen receptor signalling. Lineage-tracing demonstrates that luminal cells are favoured for organoid formation and generate basal cells in culture. Furthermore, tumour organoids can initiate from CARNs after oncogenic transformation and from mouse models of prostate cancer, and can facilitate analyses of drug response. Finally, we provide evidence supporting the feasibility of organoid studies of human prostate tissue. Our studies underscore the progenitor properties of luminal cells, and identify in vitro approaches for studying prostate biology.