Our primary objective was to investigate a biomarker driven model for the interrelationships between vascular disease pathology, amyloid pathology, and longitudinal cognitive decline in cognitively normal elderly subjects between 70 and 90 years of age. Our secondary objective was to investigate the beneficial effect of cognitive reserve on these interrelationships. We used brain amyloid-β load measured using Pittsburgh compound B positron emission tomography as a marker for amyloid pathology. White matter hyperintensities and brain infarcts were measured using fluid-attenuated inversion recovery magnetic resonance imaging as a marker for vascular pathology. We studied 393 cognitively normal elderly participants in the population-based Mayo Clinic Study of Aging who had a baseline 3 T fluid-attenuated inversion recovery magnetic resonance imaging assessment, Pittsburgh compound B positron emission tomography scan, baseline cognitive assessment, lifestyle measures, and at least one additional clinical follow-up. We classified subjects as being on the amyloid pathway if they had a global cortical amyloid-β load of ≥1.5 standard uptake value ratio and those on the vascular pathway if they had a brain infarct and/or white matter hyperintensities load ≥1.11% of total intracranial volume (which corresponds to the top 25% of white matter hyperintensities in an independent non-demented sample). We used a global cognitive z-score as a measure of cognition. We found no evidence that the presence or absence of vascular pathology influenced the presence or absence of amyloid pathology and vice versa, suggesting that the two processes seem to be independent. Baseline cognitive performance was lower in older individuals, in males, those with lower education/occupation, and those on the amyloid pathway. The rate of cognitive decline was higher in older individuals (P < 0.001) and those with amyloid (P = 0.0003) or vascular (P = 0.0037) pathologies. In those subjects with both vascular and amyloid pathologies, the effect of both pathologies on cognition was additive and not synergistic. For a 79-year-old subject, the predicted annual rate of global z-score decline was −0.02 if on neither pathway, −0.07 if on the vascular pathway, −0.08 if on the amyloid pathway and −0.13 if on both pathways. The main conclusions of this study were: (i) amyloid and vascular pathologies seem to be at least partly independent processes that both affect longitudinal cognitive trajectories adversely and are major drivers of cognitive decline in the elderly; and (ii) cognitive reserve seems to offset the deleterious effect of both pathologies on the cognitive trajectories.