Want to transfer ETH to hardware wallet but the fee is crazy expensive

I have a large sum of ETH on my hardware wallet nano s and the fee to send it to that address is 0.016 eth! That's a ridiculous sum considering atm I only want to move only 0.05 eth. What are my options? Shall I just dollar cost average more eth and send a larger sum of Eth to the address?