First step, check the two wires between the regulator & generator. No damage, connections are clean and tight.
Then I would try cleaning the contacts in the regulator. Take a piece of common newspaper, fold it once, dampen it with solvent (carb cleaner, lighter fluid, gasoline, whatever is handy) and rub it between the contacts. One set of contacts will be held closed by a spring, so open them gently, insert the paper and let the spring pressure hold them closed against the paper while you move it back and forth a few times. The other set you will have to gently hold closed against the paper.
If that doesn't work, next I would check the voltage between D and F on the control box, with the engine running at a fast idle. If the red light is on, there should be zero volts between D and F. If you find voltage here, most likely the control box is bad (given that you've already checked that the voltage is low).
For the next test, remove both wires at the generator, and connect a voltmeter from the D terminal (larger one) to ground (leaving the field open). (Contrary to what the manuals say, a digital volt or multimeter will work just fine.) Start the engine and watch the voltmeter as you increase engine rpm to 1800-2000 rpm. According to Lucas, 2-4 volts indicates the armature and brushes are OK. 0 indicates a problem with the armature or brushes; while rising voltage indicates a short between D and F.
If you get 2-4 with F open, stop the engine and connect a jumper between D and F (voltmeter still to ground). Again start the engine and watch the voltmeter as you increase engine rpm to 1800-2000 rpm. The voltage should now rise quickly as the rpm increases (don't go over 20 volts). If so, the generator is probably fine. According to Lucas, 0 volts indicates the field coils or connection (F terminal) are shorted to ground and 2-4 volts indicates an open field winding. However, I have seen generators that would generate 6-8 volts during this test, and the problem was with the armature/commutator. (Thrown solder due to overheating.)
There is more, but that will probably lead you to the problem. I recently scanned a copy of the Lucas Generator and Control Box Tests manual into a PDF file, but Basil won't let me post it here. PM me if you'd like a copy (about 2 Mb).