How does the SegWit part of a transaction get serialized and parsed?

Question

I'm reading Programming Bitcoin. And in Chapter13, I am confused with how the witness part is serialized and parsed. The auther gives the following method to parse it:

def parse_segwit(cls, s, testnet=False):
    version = little_endian_to_int(s.read(4))
    marker = s.read(2)
    if marker != b'\x00\x01':  
        raise RuntimeError('Not a segwit transaction {}'.format(marker))
    num_inputs = read_varint(s)
    inputs = []
    for _ in range(num_inputs):
        inputs.append(TxIn.parse(s))
    num_outputs = read_varint(s)
    outputs = []
    for _ in range(num_outputs):
        outputs.append(TxOut.parse(s))
    for tx_in in inputs:  
        num_items = read_varint(s)
        items = []
        for _ in range(num_items):
            item_len = read_varint(s)
            if item_len == 0:        # where I can't understand 
                items.append(0)      #
            else:
                items.append(s.read(item_len))
        tx_in.witness = items
    locktime = little_endian_to_int(s.read(4))
    return cls(version, inputs, outputs, locktime, 
               testnet=testnet, segwit=True)

And the following code to serialize it:

def serialize_segwit(self):
    result = int_to_little_endian(self.version, 4)
    result += b'\x00\x01'  
    result += encode_varint(len(self.tx_ins))
    for tx_in in self.tx_ins:
        result += tx_in.serialize()
    result += encode_varint(len(self.tx_outs))
    for tx_out in self.tx_outs:
        result += tx_out.serialize()
    for tx_in in self.tx_ins:  
        result += int_to_little_endian(len(tx_in.witness), 1)
        for item in tx_in.witness:
            if type(item) == int:
                result += int_to_little_endian(item, 1)
            else:
                result += encode_varint(len(item)) + item
    result += int_to_little_endian(self.locktime, 4)
    return result

What I can't understand are the last for loops in both methods. In my understanding, these are where segwit part get serialized and deserialized, and these codes should be in one-to-one correspondence. As they do in the first half part. But they don't match in the for loops of the last half part. I mean:

• For "num_items = read_varint(s)", there should be a "result += encode_varint(len(tx_in.witness))".
• For "item_len = read_varint(s)", there should be a "result += encode_varint(len(item))".
• And for "if item_len == 0:", I don't know what this if do and where the corresponding serialization logic is.

So is anything wrong with my understanding of the segwit part logic?

Answer 1

For "num_items = read_varint(s)", there should be a "result += encode_varint(len(tx_in.witness))".

The corresponding expression seems to be the result += int_to_little_endian(len(tx_in.witness), 1), though that's WRONG, as it only works up to length of 252 witness stack elements (for more than 252 elements, varint encoding is more than a single byte encoding). Of course, such long witness stacks are pretty rare, so this probably works almost always in practice.

For "item_len = read_varint(s)", there should be a "result += encode_varint(len(item))".

The corresponding expression seems to be the first part of result += encode_varint(len(item)) + item, which also serializes the item iteself.

And for "if item_len == 0:", I don't know what this if do and where the corresponding serialization logic is.

It appears that this code supports two ways of encoding the witness stack items: they can either be integers or byte arrays. The Bitcoin script language only has one data type (byte arrays), but there are a number of opcodes that interpret these byte arrays as numbers. The encoding for the number 0 is the empty byte array.

What this code does:

• On encoding:
  • The number 0 gets serialized as the empty array (result += int_to_little_endian(item, 1)Th). That's correct.
  • Other integers get serialized as 1-byte encodings of the respective integer. This is WRONG. For small integers, the correct encoding is two bytes long.
  • Byte arrays are serialized as varlen encoding of their length, followed by the byte array itself. This is correct.
• On decoding:
  • An encoding of the empty byte array is converted to the number 0.
  • Everything else is left as a byte array.

Overall, this looks like clumsily written that nobody should use in production. It probably works within a few simple cases (never more than 252 stack elements, and no integers other than 0 being encoded), but wouldn't pass rigorous testing.