Effective accumulation of nanoparticles (NPs) in tumors is crucial for NP-assisted cancer diagnosis and treatment. With the hypothesis that aggregation of NPs stimulated by tumor microenvironment can be utilized to enhance retention and cellular uptake of NPs in tumors, we designed a smart NP system to evaluate the effect of aggregation on NPs’ accumulation in tumor tissue. Gold nanoparticles (AuNPs, ∼16 nm) were facilely prepared by surface modification with mixed-charge zwitterionic self-assembled monolayers, which can be stable at the pH of blood and normal tissues but aggregate instantly in response to the acidic extracellular pH of solid tumors. The zwitterionic AuNPs exhibited fast, ultrasensitive, and reversible response to the pH change from pH 7.4 to pH 6.5, which enabled the AuNPs to be well dispersed at pH 7.4 with excellent stealth ability to resist uptake by macrophages, while quickly aggregating at pH 6.5, leading to greatly enhanced uptake by cancer cells. An in vivo study demonstrated that the zwitterionic AuNPs had a considerable blood half-life with much higher tumor accumulation, retention, and cellular internalization than nonsensitive PEGylated AuNPs. A preliminary photothermal tumor ablation evaluation suggested the aggregation of AuNPs can be applied to cancer NIR photothermal therapy. These results suggest that controlled aggregation of NPs sensitive to tumor microenvironment can serve as a universal strategy to enhance the retention and cellular uptake of inorganic NPs in tumors, and modifying NPs with a mixed-charge zwitterionic surface can provide an easy way to obtain stealth properties and pH-sensitivity at the same time.