Rapid infant weight gain is associated with increased abdominal adiposity, but there is no published report of the relationship of early infant growth to differences in specific adipose tissue depots in the abdomen, including visceral adipose tissue (VAT). In this study, we tested the associations of birth weight, infant weight gain, and other early life traits with VAT, abdominal subcutaneous adipose tissue (ASAT), and other body composition measures using magnetic resonance imaging (MRI) and dual‐energy X‐ray absorptiometry in middle adulthood (mean age = 46.5 years). The sample included 233 appropriate for gestational age singleton white children (114 males) enrolled in the Fels Longitudinal Study. Multivariate‐adjusted general linear models were used to test the association of infant weight gain (from 0 to 2 years), maternal BMI, gestational age, parity, maternal age, and other covariates with adulthood body composition. Compared to infants with slow weight gain, rapid weight gain was associated with elevated risk of obesity (adjusted odds ratio = 4.1, 95% confidence interval = 1.4, 11.1), higher total body fat (+7 kg, P = 0.0002), percent body fat (+5%, P = 0.0006), logVAT mass (+0.43 kg, P = 0.02), logASAT mass (+0.47 kg, P = 0.001), and percent abdominal fat (+5%, P = 0.03). There was no evidence that the increased abdominal adipose tissue was due to a preferential deposition of VAT. In conclusion, rapid infant weight gain is associated with increases in both VAT and ASAT, as well as total adiposity and the risk of obesity in middle adulthood.