Hi,
Well I'm convinced that biking is probably the only sport where you can get a versatile product that is not average everywhere and good nowhere, but instead can really shine on a wide variety of conditions.
I have been riding trail or enduro bikes since I've started riding seriously 10 years ago. In recent years I've owned a yeti asr 5 (trail bike), then a yeti sb66c ("small" enduro bike) and now a santa nomad 3 (full on enduro weapon).
I've also had the chance to test very different bikes (albeit for a very short time, like a couple runs or ride): cdale flash, intense m16.
So yeah, on those short tests I didn't liked those bikes, the 29" XC hardtail race machine was terrible to ride downhill (and yet on some mellow terrain) and I was expecting a lot more punch on the uphills. The downhill bike was more confortable than my regular bike and gave me a safe feeling on big hits (like jumps) but was quite terrible on short turns and quite less fun than any enduro bike I tried that same day (on the black track at pleney morzine).
But the real reason I believe a modern enduro bike can rule them all, is that my current bike (nomad) allows me to stay competitive (strava mostly) with locals XC racers, and at the same time allows me to ride some pretty big stuffs (like 10-15m jumps or alpine tracks). I've even done some mtb-packing in the alps with it.
So yeah I don't believe I would be a lot faster (up or downs), whatever the terrain is with a trail bike or a downhill one. Maybe a freeride or downhill bike would allow me to ride bigger stuff, but I don't think my bike is currently limiting me on that front either.
So basically the limit is not the bike but the rider.
Full disclaimer, I believe all that applies thanks to a few conditions: I am a light but fit rider ( 65kg at 176cm), my bike has a very high end build (so very light and yet pretty tough) and I'm (or believe I am) quite a decent rider (I'd say above average but no out of the ordinary either).
So light bike + fit rider = good pedaling performances, strong bike + 160mm of travel + light rider = capable to go big and get away with it.
And if I had unlimited budget, my main bike would be an aggro trail bike (santa 5010 or evil the calling) with an enduro build because my local trails aren't super difficult and I find it more fun to push a small bike's limits rather than stay in a bigger bike's confort zone. And I would still own an enduro bike to race and do some freeride sessions.