Creating Witness Signature: non-mandatory-script-verify-flag (Signature must be zero for failed CHECK(MULTI)SIG operation)

Question

So I am trying to do my own witness signatures as described in bip143. I have code that correctly verifies the example signature, so I thought I had a hold of the basic signing and verification. I have code that can verify the signatures in bip143 correctly, and using the same code I verify this output correctly, so I am not sure how to determine what is going wrong. The error that I get is

error code: -26
error message:
non-mandatory-script-verify-flag (Non-canonical DER signature)

I suspect that I am not reversing the byte order in something, but I am really not sure how to further diagnose this.

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
dhashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 2cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d549e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3
der sig: 304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3 
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a8580247304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a301210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------

This is running on my regtest. I can provide any additional info that people might think would be helpful. I have spent many hours comparing every bit possible to all the examples before posting here. I know it says non connical der sig, but my researching, other people say that error happens when many different things go wrong. Also when I compare the signature to the core signed ones they are an exact match in format with the "30440220" then 32 bytes then 0220 then the other 32 bytes. The only thing that is different is the actual sig data. Plus I get the same error when using a native connical DER output. Any help is greatly appreciated as always
-----UPDATE-----
so the problem with the above script is that mainly it is not DER encoded correctly, meaning that if s or r most MSB > 0x80 it treats it as an negative because it is a signed integer. In this example the MSB of S is 0xE8 which is greater then 0x80. bitcoin has a policy that it actually must be less than the ecdsa curve value "n". First below is the code that corrects the problem, and I can verify the signature (but the error still exist in bitcoin core). Code to invert S in python:

n=115792089237316195423570985008687907852837564279074904382605163141518161494337
s = bytearray(s_bytes)
sint = 0
cnt = 31
for bb in s:
    sint += bb << cnt*8
    cnt -= 1
if sint >= n/2:
    print("Negated")
    sint = n-sint
    s = bytearray(sint.to_bytes(32, byteorder='big'))

not to bad. And as for r the bitcoin core just re-signs until it is lower than 0x80. it is legal for it to be higher if the der encoder correctly pads the MSB). So did it the same way. below is a new transaction that has the problem corrected, though I still get the same error :-/.

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
hashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 38d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a686be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
s before: be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
Negated
s after: 4187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
der sig: 3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85802473044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d72679929201210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------

Again, help is appreciated

Answer 1

ok figured it out! after about a week :-D. I want to post this because I spent the last week learning the bitcoin core to output the intermittent values during the sighash to compare. Anyone else needing to really understand how the core works this will hopefully be really helpful. So first my specific problem: my output script was just completely incorrect, I think I got the locking and the unlocking script mixed up, which still shouldn't invalidate the signature, but then also you notice in the example the output is like:

202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac

well the first 0x19 is not part of the script public key (the way decode raw transaction displays it). So in the description where it says

If the sighash type is neither SINGLE nor NONE, hashOutputs is the double SHA256 of the serialization of all output amount (8-byte little endian) with scriptPubKey (serialized as scripts inside CTxOuts);

you hash with the length prefix 0x19. I was including the length in the overall transaction so it still decoded fine, but not in the way I hashed. Simple misunderstanding.

Below is how I built and modified the core (0.21.1) to output the values to figure this out

1. I mostly followed this example https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests, basically installed most the dependencies from that list
2. export BDB_PREFIX='/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/db4'
3. ./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --enable-debug
4. make
5. Notice the use of --enable-debug. I think this was needed to let you use gdb? I did occasionally get thread lock errors in the log file which would cause it to crash in debug mode when using the wallet api.
6. I created a separate bin folder with some sym links to the 4 main binary executables. then edited /etc/environment to append the path to that folder. then ran "source /etc/environment". This lets you run the binaries from anywhere.
7. now you can run bitcoind -regtest -daemon. From here you can go into the source code where the binaries were built and run gdb, and attach to the process.
8. Modify <bitcoin_dir>/src/script.interpreter.cpp At the top I wrote some basic function to take a few different variable types and print to hex. Apologies for the bad code, I am C person by trade, not C++. I created a separate logging file so i didn't mess with bitcoind's main log

std::string hexhalftostr(unsigned char val)
{
    if (val <= 9)
        return std::to_string(val);
    else if(val ==10)
        return "A";
    else if(val ==11)
        return "B";
    else if(val ==12)
        return "C";
    else if(val ==13)
        return "D";
    else if(val ==14)
        return "E";
    else if(val ==15)
        return "F";
    else return "";
}
void logbytes(unsigned char *data, int len)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    std::string ss = "";
    for (int i = 0;i < len;i++)
    {
        char lsb, msb;
        lsb = 0x0f & data[i];
        msb = 0x0f & (data[i] << 4);
        ss += hexhalftostr(msb) + hexhalftostr(lsb);
    }
    myfile << ss;
    myfile.close();
}
void logbigint(uint256 data)
{
    logbytes(data.data(), 32);
}
void logint(unsigned int data)
{
    unsigned char arrayOfByte[4];
    for (int i = 0; i < 4; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 4);
}
void loglong(unsigned long data)
{
    unsigned char arrayOfByte[8];
    for (int i = 0; i < 8; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 8);
}
void logtext(std::string ss)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    myfile << ss;
    myfile.close();
}

9. SignatureHash is the function where we can actually output them as they are being computed

 logtext("\nnVersion:     ");
        logint(txTo.nVersion);
        logtext("\nhashPrevouts: ");
        logbigint(hashPrevouts);
        logtext("\nhashSquence:  ");
        logbigint(hashSequence);
        logtext("\noutpoint:  ");
        logbigint(txTo.vin[nIn].prevout.hash);
        logint(txTo.vin[nIn].prevout.n);
        //can't figure out how to grab the bytes
        ////logtext("\nscriptCode:  ");
        ////logbigint(scriptCode);
        logtext("\namount:  ");
        loglong(amount);
        logtext("\nnSequence:  ");
        logint(txTo.vin[nIn].nSequence);
        logtext("\nhashOutputs:  ");
        logbigint(hashOutputs);
        logtext("\nnLockTime:  ");
        logint(txTo.nLockTime);
        logtext("\nhashType:  ");
        logint(nHashType);

And the return of the function had to be replaced because the sha256 hash consumes the data so

 return ss.GetHash();

Becomes

 uint256 sighash = ss.GetHash();
 logtext("\nhashed image: ");
 logbigint(sighash);
 logtext("\n");
 //return ss.GetHash();
 return sighash;

10. I would recomend setting a break point and stepping through "EvalScript" function. This is where all the magic happens, and is super good for educating yourself on how the actual bitcoin script runs, because each error code encompasses virtually unlimited mistakes, I am not sure there is any other way to debug your code without doing this.