Maybe, since the problem occurs when pressing the brake pedal. In normal driving, the servo has manifold vacuum gated to both sides of the big piston then, when you press on the brake pedal ambient air is gated to the 'top side' of the piston creating the 'boost.' It's conceivable the servo is allowing air back into the intake manifold somehow, though I'm not sure how, yet. The servo should have a check valve that allows 'vacuum' into the piston chamber, it should prevent this but these parts are old. How well does your servo work; i.e. how large is the braking effort?
One quick test of the servo is to stand on the brake pedal as you start the car, when the engine fires the pedal should go down an inch or so. Also, you could plug or pinch off the hose from the manifold to the booster, but I've not tried this and there may be issues I haven't considered (don't try it on the road).