Perineural invasion (PNI) has been recognized as a poor prognostic factor in several malignancies, but the definition and pathogenesis of PNI in esophageal squamous cell carcinoma (ESCC) remains to be defined. PNI was evaluated by H&E staining and S100 immunohistochemistry. The predictive value of PNI in the prognosis of ESCC patients was analyzed. PNI was evaluated in vitro and in vivo. A total of 54 specimens (17.88%) were defined as PNI-a and 99 specimens (32.78%) as PNI-b. S100 staining was superior to H&E staining for PNI detection (50.66% vs 27.15%, P < 0.001, κ = 0.506). Tumor depth (P = 0.001), tumor stage (P = 0.010), and vascular invasion (P < 0.001) were significantly associated with PNI. PIN-a and PNI-b had significant lower disease free survival (DFS) and disease specific survival (DSS) than PNI-0 patients, and the prognosis of PNI-b patients was significantly worse than PNI-a patients for DFS (P = 0.009). PNI was an independent predictor for DFS and DSS in ESCC as evaluated by univariate and multivariate analyses. ESCC cells could metastasize along the nerve in vitro and in vivo, and PNI was a dynamic process. S100 staining significantly improved the accuracy of PNI detection. PNI was associated with local recurrence and poor prognosis of ESCC patients.