Replying to Broken collar bone tips

Im sure that a shitload of people on this site have done this before so I need your experience. I broke my collar bone on saturday a week ago (12/10) and I've worn my sling for the first week. I went to the orthopedist and the dr. told me I can take my arm out when I feel comfortable and that the more I use it the better. I went pretty much the whole day today and it feels weak but doesnt hurt, should i continue doing this or should I wear my sling for the 3 weeks I was originally told?