While it is true that Bitcoin can still be physically taken away from you, it is still much harder to do that than it is for a government to confiscate your fiat stored in a bank account (as most fiat is). With fiat in a bank account, the government can simply go to your bank and order them to freeze your account and lock you out of it. There is no need for them to track you down, no need to go to your physical location, and no need to beat you up. It is very simple for the government to just freeze your bank account and thus confiscate your money.
But with Bitcoin, as you say, they need to go to your house, find your Bitcoin, and forcibly take it from you. It is much more work to do that as it involves people physically going to your home in order to physically seize the devices holding the keys to your Bitcoin.
And yes, this is not any different from having a bunch of gold or a bunch of cash. As with anything which is physically in your possession, the government can confiscate it by physical force. But those physical currencies and valuables are much less convenient to transact with, particularly online.
Furthermore, many countries have regulations regarding the amount of cash that you can be carrying across the border. They often will require you to declare it, and in some instances, the money can be seized by customs. Even when you are not crossing borders, simply having a large amount of cash or other valuables in your possession may raise the suspicion of police who may confiscate them. Bitcoin does not have similar problems (with the exception of novelty physical Bitcoin coins).
With Bitcoin, you can have all of the keys for your Bitcoin stored on a flash drive, a laptop, paper, or something else. It won't be extremely obvious that you are transporting large amounts of money. Even hardware wallets just look like random electronics and flash drives so they don't raise much suspicion. So if you are carrying some Bitcoin on your laptop, it is unlikely to be seized by customs agents or police where carrying similar amounts of cash or gold bars might be.
Lastly, there is also a philosophical form of confiscation that governments partake in - inflation. Central banks nowadays routinely issue more and more fiat currency leading to some amount of inflation. Over time, any cash that you have will have less purchasing power. If you simply save money, then over time, the government is effectively confiscating some of your money through inflation. On the other hand, Bitcoin has a known supply that eventually becomes fixed. While there is some inflation, because of the eventual fixed supply, Bitcoin largely becomes deflationary thus allowing your saved Bitcoin to actually have a greater purchasing power in the future. While precious metals currently have a fairly fixed supply, theoretically it would be possible to find large quantities of them somewhere else in the universe which would then cause the value of those precious metals to go down. With Bitcoin, that is not possible.