You cannot change this. It is a design limitation of the network. Continuing reading for an explanation.
What blockchain.info is referring to is the data size of your transaction. Bitcoin fees are based on the data size of the transaction, usually marked as a btc/byte value.
Transactions can vary a great deal in size because of the number of inputs and outputs. Any bitcoin address you have is actually made up of all the inputs it has received. If you have received 10 transactions into a single address, emptying the address will require all 10 of those inputs plus at least 1 output. This adds data size to the transaction, therefore it requires a higher fee.
Blockchain.info is telling you this is the case with your address, and it's not particularly unusual. I had an old address that served as my mining receiving address for several years. It just collected mining profits in small chunks over all that time. When I finally decided to move all of it, there were 72 inputs and one output, amounting to a massive fee. It kind of sucks, but that's one of the limitations of the network.
One thing to note is that mining fees are entirely unrelated to the value of bitcoin or the sum of the transaction outputs. Fees are only based on the data size of the transaction. A 100 bitcoin transaction with only one input and one output will be a significantly less fee than a 1 bitcoin transaction with ten inputs and ten outputs.