In the regulated secretory pathway, secretory granules (SGs) store peptide hormones that are released on demand. SGs are formed at the trans-Golgi network and must undergo a maturation process to become responsive to secretagogues. The production of mature SGs requires concentrating newly synthesized soluble content proteins in granules whose membranes contain the appropriate integral membrane proteins. The mechanisms underlying the sorting of soluble and integral membrane proteins destined for SGs from other proteins are not yet well understood. For soluble proteins, luminal pH and divalent metals can affect aggregation and interaction with surrounding membranes. The trafficking of granule membrane proteins can be controlled by both luminal and cytosolic factors. Cytosolic adaptor proteins (APs), which recognize the cytosolic domains of proteins that span the SG membrane, have been shown to play essential roles in the assembly of functional SGs. Adaptor protein 1A (AP-1A) is known to interact with specific motifs in its cargo proteins and with the clathrin heavy chain, contributing to the formation of a clathrin coat. AP-1A is present in patches on immature SG membranes, where it removes cargo and facilitates SG maturation. AP-1A recruitment to membranes can be modulated by Phosphofurin Acidic Cluster Sorting protein 1 (PACS-1), a cytosolic protein which interacts with both AP-1A and cargo that has been phosphorylated by casein kinase II. A cargo/PACS-1/AP-1A complex is necessary to drive the appropriate transport of several cargo proteins within the regulated secretory pathway. The Golgi-localized, γ-ear containing, ADP-ribosylation factor binding (GGA) family of APs serve a similar role. We review the functions of AP-1A, PACS-1, and GGAs in facilitating the retrieval of proteins from immature SGs and review examples of cargo proteins whose trafficking within the regulated secretory pathway is governed by APs.