I remember that those buttons have an expanding type clip on the back, that if pushed in far enough will expand inside the hole that's in that welded on metal circle piece that the button sets in. Maybe you're not getting it in far enough, or the wider part of the clip is not wide enough. Also, maybe the rubber behind the circle with the hole in it is preventing you from pushing it in far enough. You can always "drill" some of that away. Perhaps expanding the wider section of the clip by turning a screwdriver between the two sections will help.
The button looks like it's in far enough, so maybe just expanding the clip will cure the problem.