In most mammals, two types of adipose tissue, white and brown, are present. Both are able to store energy in the form of triacylglycerols and to hydrolyze them into free fatty acids and glycerol. Whereas white adipose tissue can provide lipids as substrates for other tissues according to the needs of the organism, brown adipose tissue will use fatty acids for heat production. Over the long term, white fat mass reflects the net balance between energy expenditure and energy intake. Even though these two parameters are highly variable during the life of an individual, most adult subjects remain relatively constant in body weight throughout their lives. This observation suggests that appetite, energy expenditure, and basal metabolic rate are linked. An important characteristic of the adipose tissue is its enormous plasticity for volume and cell-number variations and an apparent change in phenotype between the brown and white adipose tissues. The present review focuses on the cellular mechanisms participating in the plasticity of adipose tissues and their regulation by the autonomic nervous system. There is compelling evidence with regard to the importance of the nervous system in the regulation of adipose tissue mass, either brown or white, by acting on the metabolic pathways and on the plasticity (proliferation, differentiation, transdifferentiation, apoptosis) of these tissues. A better comprehension of the different mechanisms involved in the feedback loop linking the brain and these two types of adipose tissue will lead to a better understanding of the pathophysiology of various disorders including obesity, cachexia, anorexia, and type II diabetes mellitus.