its a different kind of physical toll, Ive played team sports all my life, and ive raced all my life, in a car with a decent amount of downfore, only about 2 to 2.5 cornering g's, and in karts, still nothing close to indycar or F1, id say the physical strain you put on your body is more than in something like soccer and football, both which i pretty much played on and off in elementary through highschool. if anything its more of a constant strain in racing. ive gotten more cramps, sore muscles, sore joints etc from racing than i ever did in sports. you have to resist more forces and more constantly. sure its not super hard to turn a wheel but there is a lot of resistance and to do it while under 4+g's and having to make insanely quick tiny inputs, also while braking, and doing it all with a huge amount of finesse and feel. your body is constantly more tense than in anything else ive done. that and the heat can be extremely intense. indycar drivers loose something like 10 lbs of water weight in a single race.
where something like soccer is tougher physically is in cardio, sprinting back and fourth down a field, even with the short breaks you get is pretty exhausting.
both take a pretty similar physical effort, just in different ways.
where fitness is important in racing is to assist the mental aspect. racing takes such an insane amount of mental awareness, concentration, feel, etc.. that you have to be able to ignore and not be effected by any physical strain. if your mind is distracted by a straining feeling anywhere in your body, thats concentration thats not going to your driving and can have a HUGE effect.
really you cant know what its like unless you experience it.