can i use the sign message function to sign the serialized hex for a p2sh transaction? Are the ecdsa functions used the same or are they different? It looks like the output is in a different format than a traditional tx signature? How can one use this function to sign?
You cannot. While both message and transaction signing use ECDSA and double sha256, signed messages slightly modify the message such that it is impossible to create a signature that works for that key in a transaction.
Specifically, bitcoin signed messages are prefixed with the string
Bitcoin Signed Message:\n (
\n is the newline character, not literally
n). This means that the data that is actually signed cannot be for a transaction since the data signed for a transaction cannot begin with that string. This design is intentional so that users cannot be tricked into signing a message which can be used in a transaction to take their money.
The output looks different just due to serialization. If the signatures were compatible, you would just deserialize the signed message signature and reserialize it differently. But because the signatures are not compatible, this point is moot.