Let's say Bob tells you to send him btc and he gives you a p2pkh address whose underlying 160-bit number is, say, 12345. You can't just convert that to the taproot representation of 12345 and send btc to that taproot address; Bob wouldn't receive that right?
So, effectively, for any given 160-bit number, there are 4 different addresses you can generate from it, and sending btc to one address type would indeed send only to that address type, and the other 3 would not receive it?