Tumor- or cancer-associated fibroblasts (CAFs) are one of the most abundant stromal cell types in different carcinomas and comprise a heterogeneous cell population. Classically, CAFs are assigned with pro-tumorigenic effects stimulating tumor growth and progression. More recent studies demonstrated also tumor-inhibitory effects of CAFs suggesting that tumor-residing fibroblasts exhibit a similar degree of plasticity as other stromal cell types. Reciprocal interactions with the tumor milieu and different sources of origin are emerging as two important factors underlying CAF heterogeneity. This review highlights recent advances in our understanding of CAF biology and proposes to expand the term of cellular “polarization,” previously introduced to describe different activation states of various immune cells, onto CAFs to reflect their phenotypic diversity.