allowed DER encoding format violations for signatures in bitcoin: implications for libraries?

Question

I have recently changed verification code in python-bitcointx to use libsecp256k1 library.

In the python-bitcoinlib code (that python-bitcointx is a derivative of), openssl functions were used for signature verification.

I want to be able to remove dependency on openssl in the future, and use only libsecp256k1 in python-bitcointx, but there is a problem that I noticed when I re-wrote CECKey.verify() method:

libsecp256k1 function secp256k1_ecdsa_signature_parse_der() only allows strictly valid DER-encoded signatures.

bitcoind uses ecdsa_signature_parse_der_lax() function, copied from libsecp256k1's contrib/ directory. This function is not compiled in a normal compilation cycle of libsecp256k1.

It is expected for bitcoin-focused library to be able to verify signatures that bitcoind will also verify.

To be able to verify non-strict-encoded signatures with libsecp256k1, I see 3 options:

EDIT: I might just implement two verification methods, strict and non-strict, and then use them in tests. openssl dependency will stay because of verify_nonstrict().

• get ecdsa_signature_parse_der_lax() function from libsecp256k1/contrib, add it to python-bitcointx repository, compile at python package build time, install with a package.

This is not desireable, because it adds requirement to have C compiler to build the python package. In some environments this may be inconvenient.

• re-implement ecdsa_signature_parse_der_lax() in python

I would like to avoid this, to reduce the size of the code (reduce the surface where bug may be introduced). Porting this function correctly will require to invest some time.

It will also make verification slower to some degree, because more operations will be done in python, vs calling C functions.

• do a cycle of decode-with-openssl / encode-with-openssl / decode-with-libsecp256k1

This way all encoding violations that openssl allows are also allowed by first decode-with-openssl. encode-with-openssl then creates valid encoding, and decode-with-libsecp256k1 is happy.

The problem with this approach (in addition to this decode/encode cycle) is that it will allow some format violations that bitcoind won't allow, because ecdsa_signature_parse_der_lax() only allows an arbitrary subset of violations allowed by openssl.

python-bitcoinlib was using openssl to decode, too, so I went with the later approach - decode/encode/decode. This means I cannot drop openssl dependency.

My question is: what may be the implications of this choice ?

What I can see is that when someone uses python-bitcointx or python-bitcoinlib to verify inputs in some transaction, and these inputs contain signatures that is accepted by openssl, but not by bitcoind, they might be convinced that tx is valid, but will have unpleasant surprise when they try to broadcast it.

With some protocols that require parties to store signed transactions, that might be relevant.

As this is historical behavior of python-bitcoinlib, I am inclined to just document this in the README.md as a 'gotcha', and leave it as it is now.

But maybe there is some other potential problems that could arise out of this issue, that may be more serious, and the first or second options described above should be considered ?

Answer 1

It isn't clear to me from your question what your goal is. If you want to accept all transactions and only transactions that Bitcoin would accept, libsecp256k1 alone is what you want. (You may need to use the signature normalize call if you want to match consensus acceptance instead of standardness acceptance).

If you want to validate historical signatures from the blockchain then either openssl or the code in contrib is sufficient. If you use either of these things on current transactions you will accept things that Bitcoin will reject.

We, the authors of libsecp256k1, are currently of the belief that no application should use a lax parser, except for historical compatibility. Unfortunately, no function is likely to be universally compatible with OpenSSL's parser except OpenSSL itself... but for cases where the set of signatures that you need to be compatible with is known a specific alternative lax parser might work, as is the case for Bitcoin's history and the lax parser in contrib.

The contrib lax parser isn't included in the library because it's only provably sufficient for Bitcoin's history. Someone trying to lax-parse a different set of signatures may need to modify that lax parser to accommodate the specific weird signatures that they're dealing with. We also, in general, don't think anyone should be lax parsing unless its strictly needed since it can result in surprising vulnerabilities.

As an aside, in the current versions of OpenSSL the ECDSA validator attempts to validate signature encodings by checking that the serialization is unchanged under decode and encode and it rejects signatures that aren't. This likely means they reject any signature that libsecp256k1 (with normalize) rejects. [And annoyingly, OpenSSL made this breaking change in a security update.]

Answer 2

If you want to guarantee strict DER encoded signatures which are accepted by Bitcoind then they need to be bip66 compliant(low s values(ecdsa malleability), remove leading nulls (encoding malleability)). I am not sure you are guarding for that with your decode-encode-decode cycle. These do not enforce r,s value-space constraints.

Answer 3

It was the case that python-bitcoinlib's script evaluation code did not handle SCRIPT_VERIFY_STRICTENC flag.

I had to create additional method, verify_nonstrict(), and make the scripteval use it when STRICTENC flag is not specified.

Handled in https://github.com/Simplexum/python-bitcointx/commit/d8ab8ff76de412f90480c3c68addbbb1791d3413