You should be removing the sleeve first, then the slinger, the gear and then the key. The key is what keeps the other parts turning with the crank, rather than spinning.
If you have deep grooves on the sleeve, consider replacing it or you may possibly be facing a leaking crankshaft seal.