I.e since everything is implementable in a (Turing complete) CPU then how does it matter whether a Secure Element has support for Bitcoin's cryptographic primitives - other than making signing and decryption faster.
Firstly I think this is slightly a misconception, it is important to be able to do ECC operations efficiently which is why a theoretical Secure Element (SE) with these capabilities would need to be efficient. However the point of them is not to compete with the efficiency of say a desktop processor, the point is to have a verifiably correct and discrete processor that is efficient enough do do the operations on a small device.
Would a device with a generic CPU where the software can't be changed (without losing the secrets as well) with protection against physical attacks already be "the dream hardware wallet" which doesn't support key export no matter what?
I am not sure that smart cards fit this description perfectly, they store information and require physical interaction to release secrets but are not made for generic CPU processing if that is what you are suggesting.
I want to clarify I am certainly not an expert on this topic but from what I have researched the reason a smart card (SC) is not more secure than a SE is because you are inable to secure private key operations on the SC, you can only use it to store the private data. The SE having the capability for secp256k1 (which apparently none do atm) would allow you to do signing completely airgapped from your potentially vulnerable or already exploited personal computer system or local computer network. That being said there are non-secp256k1 related private operations that can be done on the SE, which in the case of an SC would need to be performed on your personal computer system which is an increased attack surface.
Ledger for instance claims that they use the SE to generate private keys:
Secure Element Chip Protects Your Ledger From Attacks
A Secure Element is a highly advanced chip that mitigates a lot of different types of attacks. This cutting-edge chip, which is used in high-level security solutions, really sets Ledger apart as a top-end security solution for crypto assets. All of our devices use a Secure Element, which greatly enhances their security. Ledger uses them to generate and store private keys for your crypto assets.
I do not recommend using ledger in light of their recent press releases however this is an example of how a SE might be used.
Blockstream Jade adopts a different security model not based on SE which you may find to be relevant:
Instead of a secure element, Blockstream Jade uses a unique security model that allows it to remain fully open-source while also being protected from physical attacks and achieving similar (if not better) security from this potential threat - by acting as a "virtual" secure element.
The blind oracle model that Jade uses is fully open source, and is truly blind. It knows nothing about Jade wallet data, and doesn't even know the user's actual PIN. Users may use Blockstream's blind oracle to protect their wallet, or they may run their own.
From what I can surmise, an SE can be useful for certain private operations such as generating a key however it does not close the attack surface completely. An SC because it cannot do any private operations is a more open attack surface because it delegates private operations to any computer system you plug it into. An open source blind oracle model may be an effective way to decrease the attack surface when using a SC or hardware with a similar security model such as I believe Jade might fall into. However since all SEs so far are closed source it may not be a viable option for SE based wallets to implement blind oracles to protect private key operations yet.