If you do decide prying is necessary, I'd suggest using some small strips of wood to protect the plinth, and using a pair of pry bars, one on each side, so the net force is directly outward. Using a single pry bar doesn't pull straight, which increases the chances of something breaking. Of course you'll use the 4th hand to keep the pin depressed
What I found was that the plastic knobs had shrunk with time, until they were a very tight fit on the shaft. Used an X-acto knife to trim them just a bit inside, until the fit wasn't quite so tight.
Somewhere, I've read an article about modifying a small C-clamp to apply force to the pin, but I don't recall where offhand.