The development of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) vaccines and therapeutics will depend on understanding viral immunity. We studied T cell memory in 42 patients following recovery from COVID-19 (28 with mild disease and 14 with severe disease) and 16 unexposed donors, using interferon-γ-based assays with peptides spanning SARS-CoV-2 except ORF1. The breadth and magnitude of T cell responses were significantly higher in severe as compared with mild cases. Total and spike-specific T cell responses correlated with spike-specific antibody responses. We identified 41 peptides containing CD4+ and/or CD8+ epitopes, including six immunodominant regions. Six optimized CD8+ epitopes were defined, with peptide-MHC pentamer-positive cells displaying the central and effector memory phenotype. In mild cases, higher proportions of SARS-CoV-2-specific CD8+ T cells were observed. The identification of T cell responses associated with milder disease will support an understanding of protective immunity and highlights the potential of including non-spike proteins within future COVID-19 vaccine design.