Can your Metamask wallet get hacked just by connecting to a malicious website?

I see a lot of people complaining about Metamask security on Reddit and asking for wallet alternatives. Of course Metamask itself doesn't get hacked, people are talking about phishing attacks. But is it really that easy to get scammed? Don't you have to sign a transaction that gives the third party access to your funds, or some type of message that explains what you're approving? Can you lose tokens just by connecting?