How has the design of the opcode OP_CHECKTEMPLATEVERIFY evolved over its various renames?

Question

OP_CHECKTEMPLATEVERIFY (BIP 119) has gone through various iterations. I don't particularly find the renames at all interesting (e.g. OP_CHECKOUTPUTSHASHVERIFY, OP_SECURETHEBAG) but I am interested in how the design of the opcode has changed and the trade-offs for these changes. My understanding is that the changes have mainly been motivated by how flexible or how generic the opcode should be i.e. whether it should support multiple use cases or be focused on one particular use case. I also understand that recursive covenants were initially supported but are not supported by OP_CHECKTEMPLATEVERIFY.

The current code for OP_CHECKTEMPLATEVERIFY is here but this isn't the best place for documenting why certain design decisions were made. The BIP doesn't seem to cover the evolution of the opcode either though there is some rationale for the current design.

Answer 1

A brief back history on the proposal:

Covenants were -- as far as I'm aware -- first discussed online in 2013, here https://bitcointalk.org/index.php?topic=278122.0

MES presented in 2016 OP_COV https://fc16.ifca.ai/bitcoin/papers/MES16.pdf which was essentially a basic covenant system based on pattern matching.

In 2017 I gave a talk at Stanford's BPASE on multi transaction merkelized covenants video slides which introduces some of the motivations and design space for covenants. This is where the concept of congestion control was originally presented.

Following feedback from to conference,in 2018 I put together a system for emulating covenants using multiparty ECDSA pre-signing called lazuli and presented it in Tokyo at the bitcoin developer workshop. The feedback I received was that the core community was disinterested in looking at any ECDSA based designs since schnorr was imminent.

Modern History:

The Lazuli concept (as you can see from the whitepaper) formed the basis for what CTV/BIP-119 became. Rather than doing ECDSA multi-party, maybe developers would find it acceptable to just directly commit to the pre-signed transaction hash. The difficulty here was making a comitted covenant that would not permit recursive covenants -- at least not unintentionally.

In May 2019 I presented the initial draft of this covenant idea under the name CHECKOUTPUTSHASHVERIFY to SF BitDevs. Slides. I focused on the congestion control use case in particular as a clear cut motivation, although the presentation makes brief mention of other applications. Notably I included "spawning channels" as one of the intended use cases. This design also used a new pattern for a "multi byte opcode", a "forward argument" where the verify semantics apply to the next argument ensuring that the data could not come from any computation but had to be statically known at address generation time. This was to make it trivial to prove no recursive covenants were possible.

I received feedback from online and in person:

1. Russel O'Connor made a strong argument to not do a forward peeking argument, since the regular argument order (checking the thing on the stack not the script) was also safe & preserved some script composition rules.
2. Greg Maxwell pointed out in IRC that I needed to commit to more fields to prevent malleability.
3. The name was universally hated because it was so boring

In response:

1. I kept the forward peeking argument design (ignoring Russel's request) as I felt that the principal was already violated by OP_IF
2. More fields were comitted to to eliminate unintentional malleability and make it possible to predict the TXIDs in trees of transactions perfectly (for, e.g., channels).
3. It was renamed OP_SECURETHEBAG

Over the next several months, I gathered more feedback & made more tweaks on the design:

1. In more conversations with Russell, we reached agreement to drop the forward peeking argument design in favor of normal reading from the stack (creating an upgrade path for using CTV more generally with things like OP_CAT).
2. The input_index was added to the digest to preclude the "half spend problem" where 2 identical covenants could unintentionally be satisfied in a single transaction.
3. The name OP_SECURETHEBAG was both more loved and hated than COSHV. COSHV was no longer technically accurate, so the name was changed to CheckTemplateVerify.

At this juncture (roughly January 2020) I felt that the BIP was conceptually mature enough & code wise stable enough to introduce a BIP.

In February 2020 I held the utxos.org workshop to discuss the design of CTV (in it's current form).

Around April 2020, I also broke ground on Sapio (originally mentioned at Scaling Bitcoin '19 Tel Aviv as a possibility) to generalize the tooling for the smart contracts I was developing for CTV but hand rolling the code for.

Around February 2021, I released the first open source version of Sapio which had undergone many iterations (i.e., re-implemented from python to rust, custom scripts to miniscript).

Overall I don't find your assessment is quite correct as recursive covenants were never a part of CTV's design (always an anti-goal). Most of the changes made were not around flexibility, but around safety & upholding the intended properties required to allow implementing something similar to Lazuli without requiring a multiparty (ECDSA or Schnorr) trust assumption. CTV's design process was inherently targetted at carving out a safe chunk of covenants to "move the overton window" on what would be possible/safe in Core. That there's now a growing wider desire for more general covenants is fantastic.

With the removal of the forward-peeking argument, CTV could be used in combination with future changes (e.g., OP_CAT/SHASTREAM) to build covenant descriptions on the stack. Effort has gone into ensuring that the transaction digest used in CTV is easy to manipulate on the stack by ordering fields in the order of least to most likely to change ("static prefix assumption").