Well your idea is correct, as it could damage the structural integrity of the helmet, but firstly it depends how old the helmet is?, the brand? and the damage history?. Modern helmets todays are so technologically advanced that i doubt that a reasonable fall would affect its structural ability. But me personally i always buys my ski gear new as i know its free from problems and i like to know that i have owned something. But sometimes you cant afford new things and a second hand piece of equipment will be just fine.
But thats just me, you have to decided these factors for yourself :)