Elevated bone marrow adiposity (BMA) is defined as an increase in the proportion of the bone marrow (BM) cavity volume occupied by adipocytes. This can be caused by an increase in the size and/or number of adipocytes. BMA increases with age in a bone-site-specific manner. This increase may be linked to certain pathophysiological situations. Osteoporosis or compromised bone quality is frequently associated with high BMA. The involvement of BM adipocytes in bone loss may be due to commitment of mesenchymal stem cells to the adipogenic pathway rather than the osteogenic pathway. However, adipocytes may also act on their microenvironment by secreting factors with harmful effects for the bone health. Here, we review evidence that in a context of energy deficit (such as anorexia nervosa (AN) and restriction rodent models) bone alterations can occur in the absence of an increase in BMA. In severe cases, bone alterations are even associated with gelatinous BM transformation. The relationship between BMA and energy deficit and the potential regulators of this adiposity in this context are also discussed. On the basis of clinical studies and preliminary results on animal model, we propose that competition between differentiation into osteoblasts and differentiation into adipocytes might trigger bone loss at least in moderate-to-severe AN and in some calorie restriction models. Finally, some of the main questions resulting from this hypothesis are discussed.