Damage to the right fusiform face area can disrupt the ability to recognize faces, a classic example of how damage to a specialized brain region can disrupt a specialized brain function. However, similar symptoms can arise from damage to other brain regions, and face recognition is now thought to depend on a distributed brain network. The extent of this network and which regions are critical for facial recognition remains unclear. Here, we derive this network empirically based on lesion locations causing clinically significant impairments in facial recognition. Cases of acquired prosopagnosia were identified through a systematic literature search and lesion locations were mapped to a common brain atlas. The network of brain regions connected to each lesion location was identified using resting state functional connectivity from healthy participants (n = 1000), a technique termed lesion network mapping. Lesion networks were overlapped to identify connections common to lesions causing prosopagnosia. Reproducibility was assessed using split-half replication. Specificity was assessed through comparison with non-specific control lesions (n = 135) and with control lesions associated with symptoms other than prosopagnosia (n = 155). Finally, we tested whether our facial recognition network derived from clinically evident cases of prosopagnosia could predict subclinical facial agnosia in an independent lesion cohort (n = 31). Our systematic literature search identified 44 lesions causing prosopagnosia, only 29 of which intersected the right fusiform face area. However, all 44 lesion locations fell within a single brain network defined by connectivity to the right fusiform face area. Less consistent connectivity was found to other face-selective regions. Surprisingly, all 44 lesion locations were also functionally connected, through negative correlation, with regions in the left frontal cortex. This connectivity pattern was highly reproducible and specific to lesions causing prosopagnosia. Positive connectivity to the right fusiform face area and negative connectivity to left frontal regions were independent predictors of prosopagnosia and predicted subclinical facial agnosia in an independent lesion cohort. We conclude that lesions causing prosopagnosia localize to a single functionally connected brain network defined by connectivity to the right fusiform face area and to left frontal regions. Implications of these findings for models of facial recognition deficits are discussed.