Many years ago, I tried using an O2 sensor mounted into the exhaust manifold. It was the "narrow band" type, and as a result returned basically no useful information. The original SU carbs simply do not hold the mixture at the theoretical ideal mixture no matter how you adjust them; and the sensor I had (same as used in most modern cars) was not responsive either at the richer mixture that produces best power; or the leaner mixture that produces best fuel economy. But the slightest movement of the throttle would change from one extreme to the other.
As far as why, as I've written before, any change in how the engine breathes can cause a change in what needle is required to produce optimum mixture. That is why there are so many needles available, almist every car used a different needle. So bigger liners, a bigger exhaust, exhaust headers, a different cam and so on all may need a needle change for best performance.
There are also various defects, like a worn jet, that can cause the normal adjustment procedure to not produce best mixture. In retrospect, I believe that is the reason that my Dads TR3A would always overheat, and eventually swallowed a valve. It cruised so lean, and EGT was so high, that it eventually eroded an exhaust valve stem until it failed. At the time, we blamed it on the 4.1 axle and no OD, but I've since run that combination on my own TR3 and it did not overheat under the same conditions.