The emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants with mutations in key antibody epitopes has raised concerns that antigenic evolution could erode adaptive immunity elicited by prior infection or vaccination. The susceptibility of immunity to viral evolution is shaped in part by the breadth of epitopes targeted by antibodies elicited by vaccination or natural infection. To investigate how human antibody responses to vaccines are influenced by viral mutations, we used deep mutational scanning to compare the specificity of polyclonal antibodies elicited by either two doses of the mRNA-1273 COVID-19 vaccine or natural infection with SARS-CoV-2. The neutralizing activity of vaccine-elicited antibodies was more targeted to the receptor-binding domain (RBD) of the SARS-CoV-2 spike protein compared to antibodies elicited by natural infection. However, within the RBD, binding of vaccine-elicited antibodies was more broadly distributed across epitopes compared to infection-elicited antibodies. This greater binding breadth means that single RBD mutations have less impact on neutralization by vaccine sera compared to convalescent sera. Therefore, antibody immunity acquired by natural infection or different modes of vaccination may have a differing susceptibility to erosion by SARS-CoV-2 evolution.