Could, game theoretically speaking, Bitcoin work without block subsidy and transaction fees? Assume, for example, some alternative anti-transaction-spam measure was in place. Would there be enough incentive for miners to use their computing power for no direct reward other than keeping the network up? Could the network still operate safely under this scenario?
Bitcoin would have to be changed heavily for that to work.
Here are some of the issues with that:
- Initial money distribution: Currently, all new coins are introduced into the system through the block subsidy. If there is no block subsidy, how does anyone ever get any coins?
- Security against majority attack: The mining reward (consisting of block subsidy and fees) incentivizes miners to invest money into hardware and to spend irretrievable resources (power). Why would they make investments or mine at a loss, when there is no reward? Perhaps some honest users waiting for confirmation of their transactions would mine until a block is discovered and they get their transaction confirmed, and then stop again. Perhaps some people would even mine at a loss just because they can.
However, the hash rate of the network would be minuscule, and an attacker e.g. with a botnet could easily overwhelm it, and start a majority attack.
Additionally, previously attackers could usually earn more by supporting the network than by attacking it (unless they provided already a majority of the hash rate by themselves). Now, there is no reward to support the network, so an attack is always more lucrative than supporting the network.
- Transaction ordering: Currently, transactions are prioritized by their fee rate. Transactions that pay more per byte get selected for confirmation first. If transactions were free in general, people could send an unlimited amount of transactions. They could for example use the Bitcoin blockchain as cheap decentralized backup for their data. However, you say that there is some sort of anti-spam measure in place. Another problem still exists: How do you decide which transactions get confirmed first? "First come, first served" could work to some degree, but if more transactions are submitted to the network than get confirmed, waiting times for confirmations grow indefinitely.
There are some Proof-of-Stake systems that don't have a block reward. ( Instead they created all coins at genesis and distributed them in some fashion to the initial user base. However, Proof-of-Stake doesn't cost the expenditure of irretrievable resources. AFAIK, transactions in PoS systems still cost a transaction fee.