Is there a technical reason for which there isn't an option in Bitcoin Core that would deploy an internal Electrum Server, or something similar so that a software wallet can fetch addresses and balances?
Electrum servers are simply not compatible with Bitcoin Core on a development or operational level. They require a vast amount of indexes created which the node does not have, and must be created by an external tool. With the state of the art
electrs Electrum server, it needs upwards of a day of time to create these indexes, and more than 50GB of additional data stored on disk. For the server to be performant, forks of the software use over 1TB of additional storage for these indexes.
I think the preferred technique to use 'bitcoin core as a wallet backend' is through block filters, like the new compact block filters (an improvement over bloom filters). The protocol is known as Neutrino and lowers the CPU and storage requirements for your bitcoin core node (an electrumx database takes around 50 GB right now). To create these filters in bitcoin core, you need to launch add the bitcoind option
blockfilterindex=basic. With these filters, you can filter for blocks that affect the addresses (scriptPubKeys) that belong to your wallet. Several lightning wallets use this technique
This block filtering technique produces some false-positives, so you will fetch blocks that end up not affecting your addresses. Because you are requesting more than just the relevant transactions to your wallet, you end up consuming much more bandwidth than using an electrum server but it has the advantage of being more private.
Neutrino is used by several lightning network wallets: