Tumors with high mutational burden (TMB) tend to be responsive to immune checkpoint blockade (ICB) because there are neoantigens available for targeting by reinvigorated T cells, whereas those with low TMB demonstrate limited clinical responses. To determine whether antigen-specific T cell responses can be elicited after treatment with ICB in cancers that have a low TMB, we conducted a clinical trial with ipilimumab in 30 patients with metastatic castration-resistant prostate cancer. We identified two distinct cohorts by survival and progression times: “favorable” (n = 9) and “unfavorable” (n = 10). Patients in the favorable cohort had high intratumoral CD8 T cell density and IFN-γ response gene signature and/or antigen-specific T cell responses. Two patients with a relatively low TMB had T cell responses against unique neoantigens. Moreover, six of nine patients in the favorable group are still alive at the time of analysis, with survival ranging from 33 to 54 months after treatment. All 10 patients in the unfavorable cohort have succumbed to their disease and had survival ranging from 0.6 to 10.3 months. Collectively, our data indicate that immunological correlates associated with effector T cell responses are observed in patients with metastatic prostate cancer who benefit from ICB.