Thanks everyone for the explanation. So yes, only a private key can be enough to generate a bech32 address. The issue in my case seems to be that I was using
var addr = key1.ScriptPubKey.GetWitScriptAddress(net);
Key key1 = new Key();
var secret = new BitcoinSecret(key1, net);
var addr = secret.GetAddress(ScriptPubKeyType.Segwit);
Now the output ...
var bitcoinPrivateKey = new BitcoinSecret("cPof7e5g6xfgB6AZrc6XVTVwA4efLJurh9kVxa6FRChbr8Jyqaon", Network.TestNet);
var legacy_address = bitcoinPrivateKey.GetAddress(ScriptPubKeyType.Legacy);
var segwitp2sh_address = bitcoinPrivateKey.GetAddress(ScriptPubKeyType.SegwitP2SH);
var nativesegwit_address = ...