The extracellular matrix (ECM), once thought to solely provide physical support to a tissue, is a key component of a cell’s microenvironment responsible for directing cell fate and maintaining tissue specificity. It stands to reason, then, that changes in the ECM itself or in how signals from the ECM are presented to or interpreted by cells can disrupt tissue organization; the latter is a necessary step for malignant progression. In this review, we elaborate on this concept using the mammary gland as an example. We describe how the ECM directs mammary gland formation and function, and discuss how a cell’s inability to interpret these signals—whether as a result of genetic insults or physicochemical alterations in the ECM—disorganizes the gland and promotes malignancy. By restoring context and forcing cells to properly interpret these native signals, aberrant behavior can be quelled and organization re-established. Traditional imaging approaches have been a key complement to the standard biochemical, molecular, and cell biology approaches used in these studies. Utilizing imaging modalities with enhanced spatial resolution in live tissues may uncover additional means by which the ECM regulates tissue structure, on different length scales, through its pericellular organization (short-scale) and by biasing morphogenic and morphostatic gradients (long-scale).