Breakthroughs in anti-tumor immunity have led to unprecedented advances in immunotherapy, yet it is now clear that the tumor microenvironment (TME) restrains immunity. T cells must substantially increase nutrient uptake to mount a proper immune response and failure to obtain sufficient nutrients or engage the appropriate metabolic pathways can alter or prevent effector T cell differentiation and function. The TME, however, can be metabolically hostile due to insufficient vascular exchange and cancer cell metabolism that leads to hypoxia, depletion of nutrients, and accumulation of waste products. Further, inhibitory receptors present in the TME can inhibit T cell metabolism and alter T cell signaling both directly and through release of extracellular vesicles such as exosomes. This review will discuss the metabolic changes that drive T cells into different stages of their development and how the TME imposes barriers to the metabolism and activity of tumor infiltrating lymphocytes.