Let's image a scenario where I buy (or trade) BTC on a regular basis and have a large number of coins in my wallet. Of course I'd like to be able to use my coins whenever possible, exchange with friends or strangers locally. So, I move part of my cryptos to another wallet for petty purchases, exchanges, etc.
My concern is, suppose I have a transaction with a malicious person, they can trace back with relatively high certainty to my account. Now I know this is debatable as they couldn't really tell with certainty but maybe they just don't care. All it matters is they know or I may have cryptos. In this scenario I am left at their mercy. They could threaten me, there's cases of kidnapping, etc.
My question is, is there a reasonable way to go around this problem? Bear in mind I'm talking about local exchanges in a relatively small towns, where people know my face and name, not online where I could keep anonymity.