What is the difference between policy and consensus when it comes to a Bitcoin Core node validating transaction scripts? Can you give some examples of checks that are considered policy and checks that are considered consensus?
The term "consensus checks" refers to the rules that determine whether a block is valid; specifically, if a consensus check on a block (or a transaction contained within a block) fails, then the block is considered invalid. If we get to the end of the block and none of the consensus checks has failed, then the block is valid.
Policy checks are a superset of consensus checks (meaning that policy checks are in addition to consensus checks, so that passing policy checks implies passing consensus checks), and policy checks only apply to transactions that are being considered for addition to the mempool. The purpose of policy checks is generally to (a) close off DoS vectors and (b) to make future consensus changes safer to deploy, by preventing relay of transactions that would violate such future consensus changes well in advance.
You're specifically asking about transaction script checks, but note that there are many policy (and consensus) checks on transactions that occur outside of just script validation. Also, the links in your question are to the "policy/" and "consensus/" subdirectories in Bitcoin Core's source tree, but there are many policy and consensus checks that happen outside of those directories.
Here are some examples of policy checks in script validation, that are not (yet) consensus checks. These first ones are from BIP62, which proposed a set of rules to limit transaction malleability.
Here are some policy checks relating to making softfork deployment safe in the future:
There are some miscellaneous other policy script checks, such as requiring that all pubkeys in segwit scripts be compressed, or that inputs to OP_IF/OP_NOTIF be minimally encoded. And then of course all future consensus changes (such as Taproot changes) that have not yet been activated on the network are policy checks as well. An (almost) full list of what transaction script validation checks are performed as part of policy checks when a transaction is being validated for the mempool can be found here. (Note that this list includes consensus checks as well.)
However that is not quite a complete set of transaction script validation rules that are enforced as policy checks. During mempool acceptance, we perform other checks on a transaction as well, some of which depend on a transaction's scripts. Some examples of these include:
- Non-p2sh, non-segwit spends must be P2PKH, P2PK, or bare multisig
- scriptSig or witness must be no more than a given size
- Various rules on outputs created (no more than one OP_RETURN, must be P2SH/v0 segwit/P2PKH/P2PK/MULTISIG, value must be above dust threshold)
- Limits on how many signature checks are performed by the transaction
There are a lot of these to list, but tracing the code in mempool acceptance (particularly here) and inspecting functions like
CheckTxInputs should yield the remaining policy script checks, on top of the ones enforced inside the script interpreter.
Transaction script consensus checks are a much smaller set. Some of the script flags that are now consensus checks include:
The full list (with enabling conditions) can be seen here. Note that as consensus rules have changed in Bitcoin's history, not all the script flags mentioned in that function are enabled on all blocks.