Using "Sign Message" one can prove ownership of an address.
This happens by signing a human-readable string using the private key belonging to the address.
The owner then provides the address, message and signature, and anyone can verify that they indeed hold the private key of the address.
The ECDSA signature is normally provided in 65-byte format (
Recovery byte + 32-byte R + 32-byte S) and base64 encoded.
However, there is also another type of signature, which is used for "Sign Transaction".
For each input of a transaction, there is some data to be signed. This data is binary data, normally provided in hex format.
The corresponding signature is provided in 70-, 71-, or 72-byte DER format.
In both cases, the message (message string in the first case, and binary per-input data in the second case) is transformed into a byte array, and the private key is used to generate an ECDSA signature for that message byte array.
The output format is different, but the 65-byte format can easily be converted into DER.
Consequently, I thought that I could just use "Sign Message" (with a minor modification, so that it takes a hex string instead of an ASCII/UTF-8 string) to sign a transaction's hex-encoded binary per-input data.
I did that, converted the signature to DER format and tried to send the transaction, but it didn't work, because the signature was wrong.
So my question is, what is the difference between how "Sign Message" and "Sign Transaction" generate signatures, and why aren't they compatible?
Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?