Not true at all. The 4th and 5th gen are a little more "refined" than the 1st-3rd, but they're still built for off-road use. ATRAC in the newer ones is fucking AWESOME. It makes having a rear locker basically pointless unless you're crawling. And they made them up to 09 with a manual option, so check your facts.
But OP, it really depends on what you want. A WRX will be more fun to drive, but a 4Runner will get you anywhere and won't quit.