Many of the remaining challenges in infectious disease control involve pathogens that fail to elicit long-lasting immunity in their hosts. Antigenic variation is a common reason for this failure and a contributor to the complexity of vaccine design. Diversifying selection by the host immune system is commonly, and often correctly, invoked to explain antigenic variability in pathogens. However, there is a wide variety of patterns of antigenic variation across space and time, and within and between hosts, and we do not yet understand the determinants of these different patterns. This review describes five such patterns, taking as examples two bacteria (Streptococcus pneumoniae and Neisseria meningitidis), two viruses (influenza A and HIV-1), as well as the pathogens (taken as a group) for which antigenic variation is negligible. Pathogen-specific explanations for these patterns of diversity are critically evaluated, and the patterns are compared against predictions of theoretical models for antigenic diversity. Major remaining challenges are highlighted, including the identification of key protective antigens in bacteria, the design of vaccines to combat antigenic variability for viruses and the development of more systematic explanations for patterns of antigenic variation.