Cellular and humoral immunity to SARS-CoV-2 is critical to control primary infection and correlates with severity of disease. The role of SARS-CoV-2-specific T cell immunity, its relationship to antibodies, and pre-existing immunity against endemic coronaviruses (huCoV), which has been hypothesized to be protective, were investigated in 82 healthy donors (HDs), 204 recovered (RCs), and 92 active COVID-19 patients (ACs). ACs had high amounts of anti-SARS-CoV-2 nucleocapsid and spike IgG but lymphopenia and overall reduced antiviral T cell responses due to the inflammatory milieu, expression of inhibitory molecules (PD-1, Tim-3) as well as effector caspase-3, -7, and -8 activity in T cells. SARS-CoV-2-specific T cell immunity conferred by polyfunctional, mainly interferon-γ-secreting CD4⁺ T cells remained stable throughout convalescence, whereas humoral responses declined. Immune responses toward huCoV in RCs with mild disease and strong cellular SARS-CoV-2 T cell reactivity imply a protective role of pre-existing immunity against huCoV.