I understand that Bitcoin uses unspent tx output and Ethereum account balance based. However, recently there are lots of discussions on that Bitcoin's UTXO database has become too big. Does this mean account balance based transactions are better in scaling than UTXO based?
The choice between the UTXO model and the balance model is primarily one between privacy incentives and apparent intuitiveness.
If one follows the standard advice of not reusing addresses/outputs/scripts, as to not gratuitously reveal which coins belong to the sender and which belong to the receiver, the two models are actually equivalent. In this case balances would be single use anyway, and there would be as many balances as there would otherwise be UTXOs, removing both the apparent size advantage and convenience.
However, the balance model effectively incentivizes reuse. As the cost to the system (possibly in fees, but certainly in node operation costs) for a balance update is lower than the creation of a new balance, such a system inherently incentives revelation of transaction sources.
And even though balances are more intuitive, they do complicate reasoning about the security of the system. One issue is the problem of replay attacks. Assume X paid Y, and Y pays Z. Now W pays Y again. In a naive implementation, Z can rebroadcast his earlier transaction from Y, and could take the money from W's payment to Y. One solution is by putting sequence numbers on balances, and requiring transactions to list the sequence number of the transactions they take money of. Unfortunately, this requires maintaining the sequence number even for balances that went to zero, meaning that the balance database needs to maintain some data for every balance that was ever nonzero; a result that's far less scalable than the UTXO model was in the first place (which can forget outputs as soon as they're spent).
In short, my view is that the balance model reduces the fungibility of the currency as a whole for a short term gain that disappears.
In UTXO model, each transaction spends output from previous transactions and creates new output that can be spent by transactions later on operated by full nodes. Your wallet maintains a list of UTXOs associated with all addresses owned by the user, and the balance of the wallet is calculated as the sum of these unspent transactions. Take a look at this example :
A has 2.5 btc i.e. one UTXO of 2.5 btc. A wish to send B 1.0 btc. A uses 2.5 btc as input to the trxn and sends 1.0 bitcoin to B, 1.5 btc sent back to A as a new UTXO with new address. So If B has 1 btc then total is 2 btc (sent by A) and having two UTXOs: one already of 1 btc and other one is of 1 btc by A.
Bitcoin blockchain uses this model, there is another model which is being used by Ethereum blockchain i.e. Account Model which is quite simpler than the previous model. Lets see the example :
A has 2 ethers, A wish to send B 1 ether, so the network will reduce 1 ether from A i.e. A has 1 ether now and network adds 1 ether to B. If B has 2 ethers already then total is 3 ethers.
Both models has different pros and cons.
Every transaction in account model make one output which leads to space savings. Any space savings is a benefit. As the network full nodes are meant to be operated by individuals rather then data centers the lower the data requirements the more potential participants.
But UTXO provides a higher level of privacy as the users use new addresses for each transaction and will be difficult to link accounts to each other.