A longstanding unsettled question is whether pancreatic β cells originate from exocrine duct cells. We have now used genetic labeling to fate map embryonic and adult pancreatic duct cells. We show that Hnf1β⁺ cells of the trunk compartment of the early branching pancreas are precursors of acinar, duct, and endocrine lineages. Hnf1β⁺ cells subsequent form the embryonic duct epithelium, which gives rise to both ductal and endocrine lineages, but not to acinar cells. By the end of gestation, the fate of Hnf1β⁺ duct cells is further restrained. We provide compelling evidence that the ductal epithelium does not make a significant contribution to acinar or endocrine cells during neonatal growth, during a 6 month observation period, or during β cell growth triggered by ligation of the pancreatic duct or by cell-specific ablation with alloxan followed by EGF/gastrin treatment. Thus, once the ductal epithelium differentiates it has a restricted plasticity, even under regenerative settings.