I recently took a deep dive into studying Bitcoin technology, and I'll try to answer this in retrospect.
SegWit was designed within boundaries of one key constraint: you shall not hard-fork; meaning that Bitcoin consensus specification could only become more tight, not less. In other words, only soft-forking consensus changes would have been considered.
With that constraint in mind, SegWit is a solution where you'd logically arrive at if you wanted to address the two key problems at the same time:
- Block size limit
- Transaction malleability
which was the case at that time in Bitcoin history.
Block size limit
If we wanted to address just the blocksize limit then we'd have to put some data "outside" the data structure covered by the 1MB blocksize limit, have the data validated by upgraded nodes, and have a hash anchor it somewhere in the legacy 1MB block so integrity of the blockchain remains tight.
One solution would have been to just have a kind of extended block space, that could be specified with some new size limit or an algorithm that would be adjusting it. This would require solving these 2 problems:
- Moving BTC from 1MB space to the extended block space would have to be seen as a valid operation from the point-of-view of non-upgraded nodes.
- Moving BTC back from extended block space back to 1MB space would also have to be seen as valid operation from point-of-view of non-upgraded nodes.
This could be solved by declaring some anyone-can-spend locking script pattern as a magical address that will hold the record of balances moved to extension block and have all transactions that move BTC between the 2 spaces encode entry/exit points so that those migration transactions are considered valid by non-upgraded nodes. Upgraded nodes would prevent someone just taking the funds as if it was really anyone-can-spend, and non-upgraded nodes trying to steal from the address would see their transactions rejected for reasons unknown. This kind of solution was actually proposed and rejected at the time SegWit was being discussed.
It would have been more complex than SegWit for many reasons and it would introduce another key problem for non-upgraded nodes: they'd not have the correct view of the UTXO state. They'd just see this black box address where BTC sometimes goes in and comes back from, they'd have no idea of who owns the BTC inside the black box (the extended blocks space).
SegWit doesn't suffer from those problems, because it doesn't change the definition of UTXO state and it doesn't change the view of UTXO state transformations for non-upgraded nodes. They'd continue to see the correct view of WHAT has changed, but they'd not anymore have the correct view of WHY has it changed because they'd not see the "witness" anymore. If we consider "you shall not change the UTXO view" as another constraint then SegWit is the maximum possible circumvention of the "hard" 1MB limit. Any further circumvention would require some approach like described above.
If we wanted to address just the transaction malleability then the "witness" could have still been "segregated", but it wouldn't have been necessary to segregate it outside the transaction or the 1MB "hard" block space. It could have stayed inside to keep it all less complex as it would have been sufficient to segregate it just from the signature preimage, so that whole transaction except the signature itself could be covered by the signature. This would have avoided touching the Merkle tree structure, and the change would have been contained to transaction validation rules. So, solving transaction malleability alone could have been less complex than SegWit, but that was not the only objective of the upgrade.
Conclusion on SegWit
By adopting the approach of only moving the input data somewhere to solve (1.), it became almost "free" to solve (2.) so naturally that path to solving transaction malleability was taken. There are some alternative design decisions like particular choices in encoding or choice of sha256 vs sha256d that could be argued about, but on a high-level I think we can say that SegWit is the logical result of engineering within the two tight constraints: the upgrade being soft-fork only and maintaining a consistent view of UTXO set and its state transformations.
Note on anyone-can-spend FUD
When SegWit was being intensely discussed, people were bringing up the issue of possibility of someone stealing the funds if SegWit hashrate would drop, because from point-of-view of non-upgraded nodes the BTC locked in SegWit addresses would be free for the taking. This is not possible, because it is not only mining nodes which were upgraded: after activation the new consensus rules are enforced by all nodes, even non-mining nodes, and they'd straight-out reject transactions violating new rules even if it had more than 51% hash-power.
I think the only possibility of theft would've been by a reorg replaying post-activation SegWit transactions and having them mined in a pre-activation block so they'd be seen as valid by new nodes but this was mitigated by having a high activation threshold.
This is very much similar to P2SH BIP-0016, where someone could argue that all P2SH hashes with known redeem scripts would be free for the taking, which is simply not true.
Once activated, reverting the P2SH BIP-0016 or SegWit BIP-0141 would require a hard-fork. Convincing any remaining non-upgraded nodes to follow a chain that doesn't enforce new rules would require 51% hash-power but that chain wouldn't be BTC since most nodes have upgraded, so the hypothetical 51% attacker would be losing extreme amounts of money, making the attack on non-upgraded nodes practically impossible.
Note on hard-forks
If the soft-fork constraint would have been lifted, then the alternative solutions space would become bigger, and there would be opportunities for more efficient and less complex solutions at the expense of breaking backwards-compatibility with non-upgraded nodes while still being backwards-compatible with non-node software such as wallets, indexers, etc. However, I feel that discussing those is outside the scope of the question and may be considered off-topic here.