Security of the staking pool

In current PoW, when connecting to a mining pool I only give them my public address. In a PoS world, when connecting to a staking pool, I have to transfer my funds to them, correct? That gives me no confidence at all.

Why does solo staking have a such a high threshold at all?