Microglia regulate synaptic circuit remodeling and phagocytose synaptic material in the healthy brain; however, the mechanisms directing microglia to engulf specific synapses and avoid others remain unknown. Here, we demonstrate that an innate immune signaling pathway protects synapses from inappropriate removal. The expression patterns of CD47 and its receptor, SIRPα, correlated with peak pruning in the developing retinogeniculate system, and mice lacking these proteins exhibited increased microglial engulfment of retinogeniculate inputs and reduced synapse numbers in the dorsal lateral geniculate nucleus. CD47-deficient mice also displayed increased functional pruning, as measured by electrophysiology. In addition, CD47 was found to be required for neuronal activity-mediated changes in engulfment, as microglia in CD47 knockout mice failed to display preferential engulfment of less active inputs. Taken together, these results demonstrate that CD47-SIRPα signaling prevents excess microglial phagocytosis and show that molecular brakes can be regulated by activity to protect specific inputs.