Research shows a higher incidence of colorectal cancer in men. However, the molecular mechanisms for this gender disparity remain unknown. We report the roles of androgen in proliferation and differentiation of intestinal stem cells via targeting of the androgen receptor (AR) on intestinal stromal cells by negatively regulating BMP signaling. Orchidectomy (ORX) or the AR antagonist promotes expansion of intestinal epithelium but suppresses intestinal stem cell (ISC) proliferation. Conversely, the AR agonist inhibits ISC differentiation but augments proliferation in ovariectomized mice. Mechanistically, activation of the AR increases expression of BMP antagonists but lowers expression of BMP4 and Wnt antagonists in primary stromal cells, which promotes intestinal organoid growth. Interestingly, the BMP pathway inhibitor LDN-193189 reverses the ORX-induced effects. Our results highlight that stromal cells constitute the intestinal stem cell niche and provide a possible explanation for higher incidence rates of colorectal cancer in men.