Protein toxins such as ricin and Shiga toxin with intracellular targets have to be endocytosed and translocated to the cytosol to inhibit the protein synthesis and thereby kill the cell. Ricin is internalized by both clathrin-dependent and -independent endocytic mechanisms, whereas Shiga toxin seems to be taken up exclusively from clathrin-coated pits. After endocytosis, internalized membrane and content are delivered to endosomes, where sorting for further routing in the cell takes place. Toxins that remain membrane bound at low endosomal pH can be recycled to the cell surface or transcytosed in polarized epithelia. A large proportion of internalized toxin is transported to lysosomes for degradation. Most importantly, a fraction of the internalized ricin and Shiga toxin molecules is delivered to the trans-Golgi network (TGN). Shiga toxin can, in some very sensitive cells, be transported retrogradely through the Golgi cisterns all the way back to the endoplasmic reticulum (ER), and it is possible that also ricin is transported retrogradely to the ER. In this review, a cell biological overview of these intracellular transport steps is presented, and evidence is provided that the delivery to the TGN and the subsequent retrograde transport to the ER are required for optimal intoxication. Moreover, it is argued that knowledge of this transport is important for targeted drug delivery such as the application of immunotoxins in cancer therapy.