What can be changed in signed bitcoin transaction? I mean how is possible to modify transaction to change its hash, but keep transaction valid, without need to have sign private keys?
Here is the list of known sources of malleability from BIP62 (which has been withdrawn, and is no longer up to date, but does give some insight):
- DER encoded ECDSA signatures Right now, the Bitcoin reference client uses OpenSSL to validate signatures. As OpenSSL accepts more than serializations that strictly adhere to the DER standard, this is a source of malleability. Since v0.8.0, non-DER signatures are no longer relayed already.
- Non-push operations in scriptSig Any sequence of script operations in scriptSig that results in the intended data pushes, but is not just a push of that data, results in an alternative transaction with the same validity.
- Push operations in scriptSig of non-standard size type The Bitcoin scripting language has several push operators (OP_0, single-byte pushes, data pushes of up to 75 bytes, OP_PUSHDATA1, OP_PUSHDATA2, OP_PUSHDATA4). As the later ones have the same result as the former ones, they result in additional possibilities.
- Zero-padded number pushes In cases where scriptPubKey opcodes use inputs that are interpreted as numbers, they can be zero padded.
- Inherent ECDSA signature malleability ECDSA signatures themselves are already malleable: taking the negative of the number S inside (modulo the curve order) does not invalidate it.
- Superfluous scriptSig operations Adding extra data pushes at the start of scripts, which are not consumed by the corresponding scriptPubKey, is also a source of malleability.
- Inputs ignored by scripts If a scriptPubKey starts with an OP_DROP, for example, the last data push of the corresponding scriptSig will always be ignored.
- Sighash flags based masking Sighash flags can be used to ignore certain parts of a script when signing.
(1) has been fixed for all transactions by BIP66, and (2)-(6) are prevented by standardness rules in recent Bitcoin Core releases. A specific case of (7) is proposed to be fixed by BIP147.
Segregated Witness (BIP141) proposes fix all of them, but only for transactions that spend solely from segwit outputs. It does this by moving the scriptSig data to a new field (the witness) which does not influence the txid.
No! The algorithms are generated by the first block created by bitcoin so if you change the hash, the transaction will not be recognized and invalid.
TL;DR Today - nothing
In fact, there are several sources for malleability. Most of them (not all) are already fixed by changed consensus rules, others become transaction non-standard.
But. Are you an attacker or a defender?
You can create transaction which can be malleated by any person without invalidating it and without knowledge of your keys. It is easy.