Correct, Drew. The inner portion of the outer bearing should nearly fall out in your hand. Once you clean the grease out of the hub, (usually packed with WAY too much) you can see two little divots above the inner bearing race on the inboard bearing. That's where you want to position your drift to knock the bearing out, tapping on each side alternately. Flip it back over, and you'll see the same feature for the outboard bearing race. Try to lightly tap, side to side, to get the race out as squarely as possible.
Jeff