This is kinda how high school was for me. I really didn't party that much, mainly because my main group of friends weren't really that big on partying. I always wanted some more adventurous friends, especially for skiing or biking.
Things turned almost 180 degrees for me when I hit university. I met some people that I would call my best friends now, and we partied hard and went skiing/boarding a lot, among lots of other stupid shit. They were the most interesting and fun people I could have met, and I am very glad that I did. Now, I can guarantee they will be my friends for life. This past school year, I've become 110% more social, and as a result, happier. Relationships with others are what I believe to be the key to happiness.
The lesson from all of this is to find friends you enjoy hanging out with. As many people are saying, it isn't worth ditching your friends, but if those people are holding you back from living life to its fullest, find some other people that will allow you to do that. Also, don't be friends with those people simply for your reputation; be their friends cause they're fucking awesome and you want to be a part of that.
Long ass post, but I will end with this: Take advantage of every opportunity that you can to have fun. Sure you most likely will encounter what people like to call "awkward moments" when you put yourself out there, but believe me, it is WORTH IT! Be confident and outgoing, and those awesome people will appear.
Good luck brah!