depending on the type of finish you have on your bow, you could be losing and gaining nearly that much between summer and winter. my osage bows routinely gain 2-3lbs or more in the colder, drier months, lose it in summer under commercial polyamide epoxy finish. taking off 5-6lbs with sandpaper is an awful lot of sanding, but I guess you could do it, although I wouldn't recommend it because it's more difficult to sand that much and do it evenly. I would use a Bowyer's Edge or similar depth controlled finesse scraping tool. a cabinet scraper would be my second choice.
assuming tiller is correct, reducing weight is as simple as removing wood evenly from both limbs. the best way I know to do that is actually count your tool strokes. remove wood from the sides and belly. I would remove about 3lbs, then shoot several hundred arrows, check tiller and weight. you will lose a smidgen of weight from compression, and you should allow a pound or so for the finish sanding. takes a serious bunch of shooting to be certain that no more compression is taking place in a bow that is manipulated after being well used. so, yes, you can do this without causing additional set if you work carefully.