manually signing bitcoin transaction

Question

I'm trying to create a BTC transaction with one input and two outputs, one for change and one OP_RETURN to anchor data. And I'm doing all of that on my regtest backend. The raw unsigned transaction is

01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c0000000023210340165231215a98e7a32abce9d410ecd09ac505938b25f9451defa051d591ebf8acffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a408973d6b447bdda5312b1ef1b5509668672296301dd328d4e55317def98f165d200000000

I can then use the bitcoin rpc to sign it and it works, so I'm pretty certain that this part is correct. When I however try to sign it manually I get the error

500: {"result":null,"error":{"code":-26,"message":"16: mandatory-script-verify-flag-failed (Non-canonical DER signature)"},"id":2}

I've followed the steps outlined in How to redeem a basic Tx? as well as Signing a raw transaction with Python ECDSA (or OpenSSL) as well as attempted to use pybitcointools to sign the input without success.

The address and private key I use are

addr: "mmEXEzUGcMmmiLsfxxM8gB8TQSTkuR1drf"
pk:  "cTyF9pebH3kwwzUt5gzaxSDQ1DbqYfx4P1i4d1TyjtSDEeUFgYsk"

and stragely, even though the PK is the one given to me by ./bitcoin-cli -regtest dumpprivkey <addr> does not seem to match the address and I'm not sure why or what it is I need to do to it (it's in WIF format and compressed, i.e. 0x01 appended to the end

The original TX includes the scriptPubKey of the utxo. I then double sha256 the whole raw tx (shown above) and replace the scriptPubKey part with the sigScript | pubkey using the following code:

private_key, compressed_pk = wif_to_private_key(private_key_wif)
tx = bytearray.fromhex(raw_unsigned_transaction) + int(1).to_bytes(4, 'little')

double_sha256_tx = hashlib.sha256(hashlib.sha256(tx).digest()).digest()
signing_key = ecdsa.SigningKey.from_string(bytearray.fromhex(private_key), curve=ecdsa.SECP256k1)
public_key = bytearray.fromhex(privtopub(private_key_wif))
signature = signing_key.sign_digest(double_sha256_tx, sigencode=ecdsa.util.sigencode_der) + int(1).to_bytes(1, 'little')
scriptSig = to_varstr(signature) + to_varstr(public_key)

to_varstr simply prints [len]|[data] and privtopub is the function from pybitcointools as I couldn't figure out how to go from the compressed private key to the address manually. It's worth noting though that the public_key here that privtoaddr from pybitcointools does not return the correct address either which might be the root cause of the issue?

I'd be grateful for any pointers or things to try

cheers Chris

PS: another interesting aspect is that the signed transaction I generate is a lot longer than the one generated by the signtransaction rpc call:

rpc:    01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c0000000023210340165231215a98e7a32abce9d410ecd09ac505938b25f9451defa051d591ebf8acffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a40ac4f0818b683eeeaa1fbf2f508af2fc22cc814e69025152c70d7c414ebbfc30a00000000
manual: 01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c000000006b483045022100d6e538aa819f3162d5c6a0e4d9ee0323395df89e943c769d96ae939baec5c6920220083a311cb35df3c98d7a6bd0bc80d98a71fbdf8e325ba908ea4b721a4eac8bb301210343395a6e84c7f2b1d50c11f96783664a6f04d66b51befb3befcc57334e2a9abcffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a40ac4f0818b683eeeaa1fbf2f508af2fc22cc814e69025152c70d7c414ebbfc30a00000000

Answer 1

I verified the keys on https://bitcoinpaperwallet.com/bitcoinpaperwallet/generate-wallet.html?design=alt-testnet#, and choose testnet on the first tab, and then the second tab (print front). They match, so good. Then I tried to decompose your line, where you say it is the signtransaction call. This seems to me unsigned:

 TX_IN[0] Script Sig (uchar[])
  210340165231215A98E7A32ABCE9D410ECD09AC505938B25F9451DEFA051D591EBF8AC 
  ##################################################################
  ### tcls_in_sig_script.sh: decode SIG_script OPCODES from a TX ###
  ##################################################################
    21: OP_DATA_0x21:     type tag indicating LENGTH
    03: OP_INT_0x03
        40165231215A98E7:A32ABCE9D410ECD0
        9AC505938B25F945:1DEFA051D591EBF8
        AC
    * This terminates the Public Key (X9.63 COMPRESSED form)
    * corresponding bitcoin address is:
   mvVWbgb3sZeEz495eDoaem79Qi876zuG9S

whereas the second line (manual) looks good:

 TX_IN[0] Script Sig (uchar[])
  483045022100D6E538AA819F3162D5C6A0E4D9EE0323395DF89E943C769D96AE939BAEC5C6920220083A311CB35DF3C98D7A6BD0BC80D98A71FBDF8E325BA908EA4B721A4EAC8BB301210343395A6E84C7F2B1D50C11F96783664A6F04D66B51BEFB3BEFCC57334E2A9ABC 
  ##################################################################
  ### tcls_in_sig_script.sh: decode SIG_script OPCODES from a TX ###
  ##################################################################
    48: OP_DATA_0x48:     push hex 48 (decimal 72) bytes as data
    30: OP_SEQUENCE_0x30: type tag indicating SEQUENCE, begin sigscript
    45: OP_LENGTH_0x45:   length of R + S
    02: OP_INT_0x02:      type tag indicating INTEGER
    21: OP_LENGTH_0x21:   this is SIG R
        00D6E538AA819F31:62D5C6A0E4D9EE03
        23395DF89E943C76:9D96AE939BAEC5C6
        92
    02: OP_INT_0x02:      type tag indicating INTEGER
    20: OP_LENGTH_0x20:   this is SIG S
        083A311CB35DF3C9:8D7A6BD0BC80D98A
        71FBDF8E325BA908:EA4B721A4EAC8BB3
    01: OP_SIGHASHALL:    this terminates the ECDSA signature (ASN1-DER structure)
    #########################################################
    ### procedure to strictly check DER-encoded signature ###
    #########################################################
    Minimum and maximum size constraints                        - ok
    scriptsig always starts with 0x30                           - ok
    length 138 chars is less than actual sig length (144 chars) - ok
           (hex 0x45, decimal 69, 138 chars)
    length of R coordinate (66) >= 0                            - ok
    length of S coordinate (64) >= 0                            - ok
    S-Value is within scriptsig boundaries                      - ok
    Make sure the R & S length covers the entire signature      - ok
    S-value must be smaller than N/2                            - ok
    strictly check DER-encoded signature                        - ok
    #########################################################
    21: OP_DATA_0x21:     type tag indicating LENGTH
    03: OP_INT_0x03
        43395A6E84C7F2B1:D50C11F96783664A
        6F04D66B51BEFB3B:EFCC57334E2A9ABC

    * This terminates the Public Key (X9.63 COMPRESSED form)
    * corresponding bitcoin address is:
   mmEXEzUGcMmmiLsfxxM8gB8TQSTkuR1drf

so also all DER checking of the siggnature is ok. However I see the address being different, effectivly it is the output address, isn't it? Maybe you mixed up some addresses while signing manually? What is the address you are spending from?