Why is a chain code needed for entropy in HD wallets?

Question

In the documents and articles I have read, it is stated that the "chain code" in HD wallets exists to provide entropy to the derivation of child public/private keys. If I understand the process correctly, it is the following:

MyHash = HMAC-SHA512(Parent_PublicKey + Parent_ChainCode + Index)
Special_Integer = Leftmost256Bits(MyHash)
Child_ChainCode = Rightmost256Bits(MyHash)
Child_PublicKey = Parent_PublicKey + Special_Integer
Child_PrivateKey = Child_PrivateKey + Special_Integer

My question is why exactly the chain code is necessary. Even with just HMAC-SHA512(Parent_PublicKey + Index) the output should "look" very different for different indices, so is the problem that it's still mathematically close?

A couple of related questions:

1. I assume the goal is to prevent using a key to derive its children or siblings. Is that wrong and/or are there other security assurances that HD wallets seek to provide in their tree structure?

2. Is the chain code as "public" as the public key, or is it desirable to keep it private to the extent possible?

Answer 1

For reference, the original spec is located here.

A few minor adjustment to your basic equations of how HD wallet child key derivation works. (And also note that these equations only apply for non-hardened indices)

MyHash = HMAC-SHA512(Parent_PublicKey + Parent_ChainCode + Index)
Special_Integer = Leftmost256Bits(MyHash)
Child_ChainCode = Rightmost256Bits(MyHash)
Child_PublicKey = Parent_PublicKey + Special_Integer * G
Child_PrivateKey = Parent_PrivateKey + Special_Integer

I added the * G, and changed Child_PrivateKey to Parent_PrivateKey in the last equation. The * G is because you need to use Elliptic Curve multiplication before you can add to the Parent_PublicKey, which is an Elliptic Curve point.

Now let's look at how these equations would look if the chain code wasn't used.

MyHash = HMAC-SHA512(Parent_PublicKey + Index)
Special_Integer = Leftmost256Bits(MyHash)
Child_PublicKey = Parent_PublicKey + Special_Integer * G
Child_PrivateKey = Parent_PrivateKey + Special_Integer

Now, if you know the Parent_PublicKey (which is public knowledge and may be found on the blockchain for instance), you can try many values for Index and derive all of the child keys

Now assume someone got ahold (somehow) of one of your private keys for somewhere in the tree. Without the presence of a chain code, they can derive all of the private keys down the tree from the node where they have the private key. What's worse, if they can get access to the public keys higher up in the tree, they can derive the private keys as well with a little backwards-engineering of the equations (basically, Parent_PrivateKey = Child_PrivateKey - Special_Integer). This is undesirable, as it would be better if losing a single private key doesn't compromise much/any of the rest of the HD key tree.

All of this is not possible with the chain code present. So there are very good privacy and security reasons for using the chain code.

Is the chain code as "public" as the public key, or is it desirable to keep it private to the extent possible?

It's definitely desirable to keep it private, because leaking it exposes you to the issues mentioned above.