Rab5 is a small molecular weight GTP-binding protein that functions in endocytic vesicle traffic. Like other Ras-related proteins, Rab5 is prenylated on C-terminal cysteine residues, although it lacks the typical C-terminal CAAX motif (where A is any aliphatic amino acid and X is any amino acid) to direct this post-translational modification. We have investigated structural requirements for the in vitro geranylgeranylation of Rab5. Rab5N133I, a point mutant that has impaired ability to bind GTP or GDP, undergoes modification to a limited extent and at a severely reduced rate when compared to cognate Rab5. A second point mutant, Rab5Q79L, can be processed to approximately the same extent as wild-type albeit at a reduced rate. Since the latter mutation results in defective GTPase activity, these combined observations indicate that guanine nucleotide binding plays an important role in the geranylgeranylation reaction and suggest that the GDP-bound form of Rab5 is the preferred conformation for interaction with Rab prenyltransferase. This idea is supported by the finding that non-hydrolyzable GTP analogs inhibit Rab5 prenylation, while in vitro processing of both H-ras and the gamma 2 subunit of regulatory G proteins is unaffected at concentrations of guanosine 5'-O-(thiotriphosphate) (GTP gamma S) up to 400 microM. Moreover, a truncation mutant lacking the C-terminal cysteines, Rab5(1-211), serves as an inhibitor of Rab5wt geranylgeranylation when liganded with GDP but not GTP gamma S. Thus, the recognition of Rab5 as a substrate by Rab prenyltransferase involves structural elements exclusive of the C terminus and dependent upon the GDP-binding conformation of the protein.