I do not believe in this without qualification.
There are Kenyans who can run 26 miles in 2 hours and 3 minutes. Yet they probably could not generate enough upper body strength to climb a rope, or bench press more than 50 to 100lbs. Simply refining one's fitness in one specialized way does not make them the fittest. I wholeheartedly agree with you it's a great measure of fitness, just not the only measure. What is a featherweight world champion marathoner if not the epitome of a specialized athlete? In his field he has achieved the pinnacle of success- but only in that narrow area of athleticism. Why should we attach a disproportionate amount of significance to his endeavors in a specialized field while being willing to dismiss other specialized athletes in the same breath? Take for instance an Olympic powerlifter who can squat 800lbs. (363 kg)...
Is he less fit? He's every bit as specialized. Yet he probably could not run a quarter mile. If your metric is sustained output then the Kenyan is far fitter. If your metric is the ability to generate maximum power in minimum time, the Kenyan champion is not even in contention, weighing roughly the same as the powerlifter's arm.
Are we talking fitness from a survival standpoint? From the perspective of combat? The term "fittest" is in the eye of the beholder, isn't it?