Noob question about gas fee

I have some ETH in my Coinbase Wallet that I wish to buy 0xBTC for, but I have to pay the equivalent of 100$ in gas fee to buy 500$ worth of 0xBTC… Am I doing something wrong or misunderstanding something? To me it seems a bit unfair to pay 1/5 in fees.