Model organism studies have demonstrated that cell fate specification decisions play an important role in normal brain development. Their role in human neurodevelopmental disorders, however, is poorly understood, with very few examples described. The cerebellum is an excellent system to study mechanisms of cell fate specification. Although signals from the isthmic organizer are known to specify cerebellar territory along the anterior–posterior axis of the neural tube, the mechanisms establishing the cerebellar anlage along the dorsal–ventral axis are unknown. Here we show that the gene encoding pancreatic transcription factor PTF1A, which is inactivated in human patients with cerebellar agenesis, is required to segregate the cerebellum from more ventral extracerebellar fates. Using genetic fate mapping in mice, we show that in the absence of Ptf1a, cells originating in the cerebellar ventricular zone initiate a more ventral brainstem expression program, including LIM homeobox transcription factor 1 beta and T-cell leukemia homeobox 3. Misspecified cells exit the cerebellar anlage and contribute to the adjacent brainstem or die, leading to cerebellar agenesis in Ptf1a mutants. Our data identify Ptf1a as the first gene involved in the segregation of the cerebellum from the more ventral brainstem. Further, we propose that cerebellar agenesis represents a new, dorsal-to-ventral, cell fate misspecification phenotype in humans.