Prostate cancer tissue is composed of both cancer cells and host cells. The milieu of host components that compose the tumor is termed the tumor microenvironment (TME). Host cells can be those derived from the tissue in which the tumor originates (e.g., fibroblasts and endothelial cells) or those recruited, through chemotactic or other factors, to the tumor (e.g., circulating immune cells). Some immune cells are key players in the TME and represent a large proportion of non-tumor cells found within the tumor. Immune cells can have both anti-tumor and pro-tumor activity. In addition, crosstalk between prostate cancer cells and immune cells affects immune cell functions. In this review, we focus on immune cells and cytokines that contribute to tumor progression. We discuss T-regulatory and T helper 17 cells and macrophages as key modulators in prostate cancer progression. In addition, we discuss the roles of interleukin-6 and receptor activator of nuclear factor kappa-B ligand in modulating prostate cancer progression. This review highlights the concept that immune cells and cytokines offer a potentially promising target for prostate cancer therapy.