It's been a while since I've done this, but the diagram I'm looking at from Victoria clearly shows no direct bolt linking the over-rider assembly to the bumper, as opposed to the front which does. There's no doubt you can remove the rubber and its external bracket without removing the bumper however, it may be difficult, or impossible to slip the inner bracket around the bumper. In that case, yes, you would have to remove the bumper.
I'll say this again, use two people when removing the bumper, if it slips and hits the car, you'll be very sorry you began the exercise.