2

There is a json for unit tests of Japanese characters which I want to validate using Python, specifically with this fork of pybitcointools, which has bip39 functionality.

Unit tests from Trezor's python-mnemonic test vectors work fine (in Python 2.7 IME), however, this is straightforward since there's no normalization of unicode dialectics and such, since all mnemonics are lower case English.

The Japanese fields are:

  1. Entropy (hex)
  2. Mnemonic (Japanese)
  3. Password (Japanese, appears to be the same for all tests)
  4. Seed (hex, 64 bytes)
  5. xprv

So entropy seeds mnemonic (bip39?), then mnemonic | password hashes to Seed; Seed then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)

So, assuming it's that straightforward...

  1. how is the Japanese unicode text "normalized"? (Is it just NKFD Unicode normalization, which Electrum 2.0 does?)
  2. what does "normal" mean for Japanese?
Improve this question

1 Answer 1

Reset to default
2

So entropy seeds mnemonic (bip39?), then mnemonic | password hashes to Seed; Seed then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)

That sounds about right. Most of the process is well detailed in BIP-39.

  1. An SHA-256 is taken of the entropy, and the first entropy_len_in_bits / 32 bits of this hash are appended to the end of the entropy. The resulting entropy bit string is divisible into 11-bit-long chunks (it's no longer an integral number of bytes).
  2. Each 11-bit chunk is converted into one of 211 mnemonic words.
  3. The words are joined by spaces. For display purposes in Japanese, these should be Unicode IDEOGRAPHIC SPACEs, '\u3000'. If there's no need to display the mnemonic to the user, they can be "normal" SPACEs ('\u0020').
  4. The mnemonic sentence is Unicode normalized in NFKD form. This converts any IDEOGRAPHIC SPACEs into SPACEs. It also changes some characters in some of the mnemonic words, therefore this step cannot be skipped. (The question What is NFKD normalization? is a whole separate topic that's probably best asked elsewhere IMO....)
  5. The mnemonic sentence is converted into bytes via UTF-8 encoding.
  6. The binary seed is computed as PBKDF2HMACSHA512(key= "mnemonic" | passphrase, data=utf8_mnemonic, iterations=2048, out_bytes_length=64). The passphrase can be the empty string. It must first go through the same steps 4 and 5 as the mnemonic.
  7. (this part isn't detailed anywhere AFAIK) The master extended private key is constructed by using the first 32 bytes of the binary seed as the private key, and the last 32 bytes as the chaincode.

Is it just NKFD Unicode normalization, which Electrum 2.0 does?

Electrum 2.x does use NFKD normalization, but it also performs additional steps, such as removing spaces between Japanese words after step 4. It also uses a different key string in step 6, and a completely different process prior to step 4. See this answer for an implementation of Electrum 2.x's mnemonic-words-to-seed procedure in Python.

Improve this answer
12
  • Thanks for the response. I've got gaps in how UTF-8 fits in, though I understand pretty well what NKFD does. Why not just encode Unicode directly? Also, I do understand the ideographic space, I think (\u3000 is the Japanese "equivalent" of \u0020). It was actually Electrum 2.0's implementation of the seed preparation which was confusing me. That really is strange Electrum deviated in such obscure places
    – Wizard Of Ozzie
    Jun 4, 2015 at 11:18
  • Scratch that. You can't encode Unicode 4 byte code points as well as UTF8, right?
    – Wizard Of Ozzie
    Jun 4, 2015 at 11:22
  • @WizardOfOzzie Unicode strings are simply sequences of integers in the range [0,0x10FFFF]. Before you can hash it, you need to convert it into a sequence of bytes. A simple way is to just take each 4-byte int, and use those bytes as-is (UTF-32LE encoding), but this is inneficient from a space point of view (considering that English only needs 1 byte per character). UTF-8 is more complicated, but more space efficient most of the time.
    – Christopher Gurnee
    Jun 4, 2015 at 11:27
  • @WizardOfOzzie Agreed that Electrum 2.x's normalization is much more complex, but it's helpful to minimize the chance of loss from mistyped mnemonics given the lack of any specific wordlists. (And BIP-39's requirement for specific wordlists was something Electrum 2.x's dev really disagreed with.)
    – Christopher Gurnee
    Jun 4, 2015 at 11:34
  • Does this look right? norm = lambda d: (' '.join(unicodedata.('NFKD', unicode(d)).split('\u3000'))).encode('utf-8')
    – Wizard Of Ozzie
    Jun 4, 2015 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.