The ventral pancreas, biliary system, and liver arise from the posterior ventral foregut, but the cell-intrinsic pathway by which these organ lineages are separated is not known. Here we show that the extrahepatobiliary system shares a common origin with the ventral pancreas and not the liver, as previously thought. These pancreatobiliary progenitor cells coexpress the transcription factors PDX1 and SOX17 at E8.5 and their segregation into a PDX1+ ventral pancreas and a SOX17+ biliary primordium is Sox17-dependent. Deletion of Sox17 at E8.5 results in the loss of biliary structures and ectopic pancreatic tissue in the liver bud and common duct, while Sox17 overexpression suppresses pancreas development and promotes ectopic biliary-like tissue throughout the PDX1+ domain. Restricting SOX17+ biliary progenitor cells to the ventral region of the gut requires the notch effector Hes1. Our results highlight the role of Sox17 and Hes1 in patterning and morphogenetic segregation of ventral foregut lineages.