Difference between Sign Message and Sign Transaction

Question

Using "Sign Message" one can prove ownership of an address.
This happens by signing a human-readable string using the private key belonging to the address.
The owner then provides the address, message and signature, and anyone can verify that they indeed hold the private key of the address.
The ECDSA signature is normally provided in 65-byte format (Recovery byte + 32-byte R + 32-byte S) and base64 encoded.

However, there is also another type of signature, which is used for "Sign Transaction".
For each input of a transaction, there is some data to be signed. This data is binary data, normally provided in hex format.
The corresponding signature is provided in 70-, 71-, or 72-byte DER format.

In both cases, the message (message string in the first case, and binary per-input data in the second case) is transformed into a byte array, and the private key is used to generate an ECDSA signature for that message byte array.
The output format is different, but the 65-byte format can easily be converted into DER.

Consequently, I thought that I could just use "Sign Message" (with a minor modification, so that it takes a hex string instead of an ASCII/UTF-8 string) to sign a transaction's hex-encoded binary per-input data.
I did that, converted the signature to DER format and tried to send the transaction, but it didn't work, because the signature was wrong.

So my question is, what is the difference between how "Sign Message" and "Sign Transaction" generate signatures, and why aren't they compatible?

Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?

Thank you!

Answer 1

Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?

No. It has nothing to do with the hashing. Furthermore, the input is not already a digest, it must be hashed before signing. Both the message and the input data are hashed twice with SHA256.

The difference between message signing and transaction input signing is that message signing prepends the string Bitcoin Signed Message:\n (with \n being a newline character, not literally \n) to the message before it is hashed.

Additionally, the 65 byte signature produced by message signing has a byte at the beginning that is not part of the cryptographic signature itself. Instead it is a recovery id so that the correct public key can be recovered from the signature during signed message verification. In transaction signatures, there is a different byte appended to the end of the signature which is the sighash type, usually it is the byte 0x01.

Answer 2

@andrew provided helpful information that a major difference between "Sign Message" and "Sign Transaction" is that "Sign Message" internally prepends a custom string to the textual message being signed.
Also, his answer addresses the case when you do all steps of the TX creation and signing.

I tried to keep my initial question generic, but now I see that I should have mentioned that the per-input TX data I want to sign is BlockCypher's toSign array.
In their documentation it's not immediately clear, but with some investigation, I've found out that the provided toSign hex strings are already hashed twice using SHA256, so they indeed just need to be signed.
Probably there are also other systems that provide such digests for signing.

So, in summary, if you are using an external system for creating TXs, which provides you data to sign with your private key, that data might already be hashed. One good indicator for such a digest is that it is 32 bytes long.
There are multiple reasons why "Sign Message" will not work out-of-the-box for signing such TX data:

• The TX data is binary data (usually in hex format), and "Sign Message" expects ASCII/UTF-8 strings.
• "Sign Message" internally adds a custom prefix to the message being signed.
• The TX data is already hashed, so shouldn't be hashed again.
• The output of "Sign Message" is a 65-byte ECDSA signature, and it's not in DER format (can easily be converted).

However, the actual signing is the same for both "Sign Message" and "Sign Transaction".

Hopefully this will save some research time for others with a similar use-case.