Downsides of Proof of Stake?

What are the downsides of Proof of stake vs proof of work? I'd like to ask in r/Bitcoin but it seems blasphemy to even ask that question in that echo chamber, so I ask it here because Ethereum has both and is migrating towards PoS.

As we all know Proof of Work is not environmentally completely optimal, though Bitcoin might still have an environmentally positive net effect on the environment due to less spending of a deflationary asset in the economy. Proof of Work and lack of throughput are the only things I don't like about Bitcoin. It seems Ethereum aims to solve both these issues through Proof of Stake and sharding.

So to the question. Why don't everybody use Proof of Stake when it's environmentally superior? There must be downsides to it otherwise the choice would be given. Please keep a good tone.