The 3 speed was a reliable gearbox, but only having 3 ratios it was never going to make good use of power or fuel. Torque converter lock-up was a fairly distant dream and gearchanges are fairly pedestrian. It works, but most of the engine's efforts are converted into heating up the gearbox oil.
The 4 speed was a newer gearbox and improved performance a fair bit, having 33% more gear ratios to use. Cruise fuel consumption improved too - and on a car that can manage high teens to the gallon crusing at steady speed that matters!
The V12 was originally designed to get Jaguar back into motorsport in the late 1960s. The prototype race engines had a capacity of 4.7 litres and quad cams. With some development they broke the 500bhp barrier. For road use, the bore was increased making the engine very oversquare and resulting in a capacity of 5.3 litre. This made the bottom end more suited for high RPM work, but the heads were simplified to SOHC and 2 valves per cylinder with a flat head and dished pistons. These are quite tunable either in carburreted and early injection forms. Most of the people who race E-Types and XJ-Ss in the UK prefer flat head engines which can take much bigger valves than standard. The injection system was first mass produced system with lambda sensor closed loop feedback - Jaguar were desperate to reduce the fuel consumption in the midst of the 1970s fuel crisis when prices soared.
Later, a new crankshaft was produced stroking the 5.3 litre cylinders up to 6.0 litre capacity, increasing power to 333bhp. These were the ones which got the 4 speed auto.
The Jaguar V12 is immensely strong and hugely tunable, but for megabucks. It's still the engine that Lister races at Le Mans, and it's even powered a full scale replica Spitfire in the late 1980s. Parts are readily available to increase capacity to 8.0 litres, and it's still reliable. It can be tuned to produce over 800bhp in naturally aspirated form if your pockets are deep enough. Mine aren't. Rob Beere and Wolf Racing build race engines and sell them for over £30k each.
I do still fancy a 6.0 litre of my own, but I do think I'd do the manual gearbox conversion. I like autos, but I'm used to modern 6 speed autos now and an old 1970s design churning away like an egg whisk just wouldn't cut the mustard. A good 5 or 6 speed manual will knock well over a second (closer to 2) off the 0-60 time compared to the 3 speed, and a bit less than the 4 speed whilst improving fuel consumption by around 30%.
I probably would renew all the bushes and reduce PAS assistance just to make it as tight as it would have been new, but I wouldn't spend too much effort "improving" beyond that. Rear anti-roll bars were deleted early into production although the attachments remain. These sharpen up handling a lot, so that would probably be my only deviation from "standard" other than perhaps a tasteful set of 17" alloys.