Having previously owned a TR6 for 4 years and now a TR4a for 10 years I can tell you my opinion, though others may differ....
1) The 6 cylinder engine has more pull at low revs and is smoother, but the car is no faster accelerating through the gears.
2)Assuming both cars either do or don't have overdrive there is no difference in fatigue, I have driven 400 miles in a day in both cars.
BUT, the TR4 and earlier cars have different seats, I put a folded towel behind the small of my back for back support. Also my seats don't recline, the TR6 seats do.
3)The early cars have sidescreens instead of windows and flat windscreens so more buffeting and wind noise. You often see these cars with wind wings added to reduce these factors.
4)You asked about dide, the independent rear suspension models are more comfortable on tar strips or other bumps. handling is a matter of personal opinion but I think my TR4a feels more sporty to drive than a TR6 because it has less understeer.
There really isn't a "wrong" choice, just a matter of what aspects of owning these cars appeals more strongly to you.
Suggest you go down to the nearest Triumph dealer and go for a test drive!

Simon.