Phase images obtained with gradient echo MRI provide image contrast distinct from T1- and T2-weighted images. It is commonly assumed that the local contribution to MRI signal phase directly relates to local bulk tissue magnetic susceptibility. Here, we use Maxwell’s equations and Monte Carlo simulations to provide theoretical background to the hypothesis that the local contribution to MRI signal phase does not depend on tissue bulk magnetic susceptibility but tissue magnetic architecture—distribution of magnetic susceptibility inclusions (lipids, proteins, iron, etc.) at the cellular and subcellular levels. Specifically, we show that the regular longitudinal structures forming cylindrical axons (myelin sheaths and neurofilaments) can be locally invisible in phase images. Contrary to an expectation that the phase contrast in multiple sclerosis lesions should always increase in degree along with worsening of lesion severity (which happens for all known MR magnitude-based contrast mechanisms), we show that phase contrast can actually disappear with extreme tissue destruction. We also show that the phase contrast in multiple sclerosis lesions could be altered without loss of nervous system tissue, which happens in mild injury to the myelin sheaths or axonal neurofilaments. Moreover, we predict that the sign of phase contrast in multiple sclerosis lesions indicates the predominant type of tissue injury—myelin damage (positive sign) vs. axonal neurofilament damage (negative sign). Therefore, our theoretical and experimental results shed light on understanding the relationship between gradient echo MRI signal phase and multiple sclerosis pathology.