there's a ton of things that take a long time to really become good at... that's not exactly a legitimate argument.
i agree that golf's a sport, though. it isn't so much being physically fit as much as knowing how to move your body, having great hand-eye coordination, and being able to separate yourself, mentally, from the pressure of the situation... all of which are required by every other sport. if golf isn't a sport, pitchers aren't athletes- all golfers do is hit a little white ball and all pitchers do is throw a little white ball.