B cells are increasingly recognized as playing an important role in the ongoing control of hepatitis B virus (HBV). The development of antibodies against the viral surface antigen (HBV surface antigen [HBsAgs]) constitutes the hallmark of resolution of acute infection and is a therapeutic goal for functional cure of chronic HBV (CHB). We characterized B cells directly ex vivo from the blood and liver of patients with CHB to investigate constraints on their antiviral potential. Unexpectedly, we found that HBsAg-specific B cells persisted in the blood and liver of many patients with CHB and were enriched for T-bet, a signature of antiviral potential in B cells. However, purified, differentiated HBsAg-specific B cells from patients with CHB had defective antibody production, consistent with undetectable anti-HBs antibodies in vivo. HBsAg-specific and global B cells had an accumulation of CD21^–CD27^– atypical memory B cells (atMBC) with high expression of inhibitory receptors, including PD-1. These atMBC demonstrated altered signaling, homing, differentiation into antibody-producing cells, survival, and antiviral/proinflammatory cytokine production that could be partially rescued by PD-1 blockade. Analysis of B cells within healthy and HBV-infected livers implicated the combination of this tolerogenic niche and HBV infection in driving PD-1^hiatMBC and impairing B cell immunity.