The mammary gland is composed of a diverse array of cell types that form intricate interaction networks essential for its normal development and physiologic function. Abnormalities in these interactions play an important role throughout different stages of tumorigenesis. Branching ducts and alveoli are lined by an inner layer of secretory luminal epithelial cells that produce milk during lactation and are surrounded by contractile myoepithelial cells and basement membrane. The surrounding stroma comprised of extracellular matrix and various cell types including fibroblasts, endothelial cells, and infiltrating leukocytes not only provides a scaffold for the organ, but also regulates mammary epithelial cell function via paracrine, physical, and hormonal interactions. With rare exceptions breast tumors initiate in the epithelial compartment and in their initial phases are confined to the ducts but this barrier brakes down with invasive progression because of a combination of signals emitted by tumor epithelial and various stromal cells. In this article, we overview the importance of cellular interactions and microenvironmental signals in mammary gland development and cancer.