Adipocytes store triglyceride during periods of nutritional affluence and release free fatty acids during fasting through coordinated cycles of lipogenesis and lipolysis. While much is known about the acute regulation of these processes during fasting and feeding, less is understood about the transcriptional basis by which adipocytes control lipid handling. Here, we show that interferon regulatory factor 4 (IRF4) is a critical determinant of the transcriptional response to nutrient availability in adipocytes. Fasting induces IRF4 in an insulin- and FoxO1-dependent manner. IRF4 is required for lipolysis, at least in part due to direct effects on the expression of adipocyte triglyceride lipase and hormone-sensitive lipase. Conversely, reduction of IRF4 enhances lipid synthesis. Mice lacking adipocyte IRF4 exhibit increased adiposity and deficient lipolysis. These studies establish a link between IRF4 and the disposition of calories in adipose tissue, with consequences for systemic metabolic homeostasis.