Sexual dimorphisms in the brain underlie behavioral sex differences, but the function of individual sexually dimorphic neuronal populations is poorly understood. Neuronal sexual dimorphisms typically represent quantitative differences in cell number, gene expression, or other features, and it is unknown whether these dimorphisms control sex-typical behavior exclusively in one sex or in both sexes. The progesterone receptor (PR) controls female sexual behavior, and we find many sex differences in number, distribution, or projections of PR-expressing neurons in the adult mouse brain. Using a genetic strategy we developed, we have ablated one such dimorphic PR-expressing neuronal population located in the ventromedial hypothalamus (VMH). Ablation of these neurons in females greatly diminishes sexual receptivity. Strikingly, the corresponding ablation in males reduces mating and aggression. Our findings reveal the functions of a molecularly defined, sexually dimorphic neuronal population in the brain. Moreover, we show that sexually dimorphic neurons can control distinct sex-typical behaviors in both sexes.