The experience of strong phantom limb pain (PLP) in arm amputees was previously shown to be associated with structural neural plasticity in parts of the cortex that belong to dorsal and ventral visual streams. It has been speculated that this plasticity results from the extensive use of a functional prosthesis which is associated with increased visual feedback to control the artificial hand. To test this hypothesis, we reanalyzed data of cortical volumes of 21 upper limb amputees and tested the association between the amount of use of the hand prosthesis and cortical volume plasticity. On the behavioral level, we found no relation between PLP and the amount of prosthesis use for the whole patient group. However, by subdividing the patient group into patients with strong PLP and those with low to medium PLP, stronger pain was significantly associated with less prosthesis use whereas the group with low PLP did not show such an association. Most plasticity of cortical volume was identified within the dorsal stream. The more the patients that suffered from strong PLP used their prosthesis, the smaller was the volume of their posterior parietal cortex. Our data indicate a relationship between prosthesis use and cortical plasticity of the visual stream. This plasticity might present a brain adaptation process to new movement and coordination patterns needed to guide an artificial hand.