Is centralization that scary?

Ok we talk a lot about the risk of centralization, but what about its cost? Monopoly? Even if we have only 1 miner/validator, can't people simply change blockchain? Like what is the cost of centralization? Obviously the miner/validator doesn't want to misbehave and risk destroying his wealth. We use this argument in any blockchain anyway, so why do we care about decentralization? Can we have an estimate of the cost of centralization?