Antiretroviral therapy (ART) for HIV is not a cure. However, recent studies suggest that ART, initiated early during primary infection, may induce post-treatment control (PTC) of HIV infection with HIV RNA maintained at <50 copies per mL. We investigate the hypothesis that ART initiated early during primary infection permits PTC by limiting the size of the latent reservoir, which, if small enough at treatment termination, may allow the adaptive immune response to prevent viral rebound (VR) and control infection. We use a mathematical model of within host HIV dynamics to capture interactions among target cells, productively infected cells, latently infected cells, virus, and cytotoxic T lymphocytes (CTLs). Analysis of our model reveals a range in CTL response strengths where a patient may show either VR or PTC, depending on the size of the latent reservoir at treatment termination. Below this range, patients will always rebound, whereas above this range, patients are predicted to behave like elite controllers. Using data on latent reservoir sizes in patients treated during primary infection, we also predict population-level VR times for noncontrollers consistent with observations.