If you pulled hard enough to get the shaft out, one of two things happened. Either the threaded shaft broke, or the bakelite contact reamed out too deep. If the shaft stripped, you are likely out of luck in fixing it.
If the bakelite reamed, you may be able to fix it by using a small, thin washer under the nut to catch the contact. I had to do this, as I had the same symptom as you did...over time the bakelite wallowed out and the contact would not withdraw with the spring in Scott's photo. The result...the starter keeps starting. Talk about a problem! Had to rip wires behind the dash till I got one loose to stop the spinning!
Any way...good luck.
John