The option to reduce the 49# bow to 46# is not going to go away, so I wouldn’t jump into doing it right now. It would be better to shoot some lower poundage bows for a while, and see how you feel. If you feel like working your way up to 46#, then reduce the weight. If not, sell the bow at 49#. A bow always is worth more if it hasn't been modified, plus you would save the cost of reducing the weight.
In answer to your question, a 3# weight reduction is very noticeable if it brings the bow from being a little too heavy to comfortable. OTOH, if it brings the weight from way too heavy to just too heavy, it would be a waste of money.