Proteins encoding phosphotyrosine binding (PTB) domains function as adaptors or scaffolds to organize the signaling complexes involved in wide-ranging physiological processes including neural development, immunity, tissue homeostasis and cell growth. There are more than 200 proteins in eukaryotes and nearly 60 human proteins having PTB domains. Six PTB domain encoded proteins have been found to have mutations that contribute to inherited human diseases including familial stroke, hypercholesteremia, coronary artery disease, Alzheimer's disease and diabetes, demonstrating the importance of PTB scaffold proteins in organizing critical signaling complexes. PTB domains bind both peptides and headgroups of phosphatidylinositides, utilizing two distinct binding motifs to mediate spatial organization and localization within cells. The structure of PTB domains confers specificity for binding peptides having a NPXY motif with differing requirements for phosphorylation of the tyrosine within this recognition sequence. In this review, we use structural, evolutionary and functional analysis to divide PTB domains into three groups represented by phosphotyrosine-dependent Shc-like, phosphotyrosine-dependent IRS-like and phosphotyrosine-independent Dab-like PTBs, with the Dab-like PTB domains representing nearly 75% of proteins encoding PTB domains. In addition, we further define the binding characteristics of the cognate ligands for each group of PTB domains. The signaling complexes organized by PTB domain encoded proteins are largely unknown and represents an important challenge in systems biology for the future.