Adipose tissue is a central metabolic organ. Unlike other organs, adipose tissue is compartmentalized into individual depots and distributed throughout the body. These different adipose depots show major functional differences and risk associations for developing metabolic syndrome. Recent advances in lineage tracing demonstrate that individual adipose depots are composed of adipocytes that are derived from distinct precursor populations, giving rise to different populations of energy-storing white adipocytes. Moreover, distinct lineages of energy-dissipating brown and beige adipocytes exist in discrete depots or within white adipose tissue depots. In this Review, we discuss developmental and functional heterogeneity, as well as sexual dimorphism, between and within individual adipose tissue depots. We highlight current data relating to the differences between subcutaneous and visceral white adipose tissue in the development of metabolic dysfunction, with special emphasis on adipose tissue expansion and remodeling of the extracellular matrix. Moreover, we provide a detailed overview of adipose tissue development as well as the consensus and controversies relating to adult adipocyte precursor populations.