Transcriptional activation of estrogen receptor α (ERα) is regulated by the ligand-dependent activation function 2 and the constitutively active N-terminal activation function 1. To identify ERα N-terminal-specific coregulators, we screened a breast cDNA library by T7 phage display and isolated histone deacetylase 4 (HDAC4). HDAC4 interacts with the ERα N terminus both in vitro and in vivo. Presence of the ERα DNA binding domain and hinge region reduce HDAC4 recruitment whereas full-length ERα enhances recruitment. HDAC4 interaction is selective for the ERα and not ERβ N terminus and occurs in the nucleus. We demonstrate in vivo that HDAC4 is recruited by the N terminus to the promoter of an endogenous estrogen responsive gene. HDAC4 suppresses transcriptional activation of ERα by estrogen and selective ER modulators (SERMs) such as tamoxifen in a cell type-dependent manner. Consistently, silencing of HDAC4 promotes the agonist effect of SERMs (tamoxifen and raloxifene) in a cell type-specific manner. These findings indicate a role for HDAC4 in regulating ERα activity as a novel N-terminal coregulator and uncover a mechanism by which certain cell types regulate SERM behavior.