One of my buds grew up a weekend warrior snowboarder from Atlanta. When he was 18 or 19 he moved to Taos and switched because at the time, they didn't allow snowboarding.
He is I believe...25?, and he is a ripping skier. He is on his way to becoming a back country guide, tons of avy education, and you'd never know that he had only started skiing a few years prior. He even breaks out the board once and a while, and digs both. Hell, I work with a few guys and gals that are rippers at both. Let it rip, man! It's just another challenge and something to work at. It'll only make you a better athlete.