Prefixes

Kobo dictionaries are sharded by a prefix derived from the headword.

The information in this document is based on reverse engineering DictionaryParser::htmlForWord.

Note: Kobo will only look in the file matching the word’s prefix, so if a variant has a different prefix, it must be duplicated into each matching file (note that duplicate words aren’t an issue).

Note: This document only covers the algorithm used for non-Japanese (Kanji) dictionaries.

Prefix algorithm

Prefixes are calculated using the following steps. Note that “character” refers to a single Unicode code point, not a byte.

1. Trim the word at the first null byte, if any (i.e. treat it as a C string).
2. Discard everything but the first two characters.
3. Convert the characters to lowercase using the Unicode case mapping rules.
4. Trim all whitespace characters on the left and right sides.
5. If the string is empty, return “11”.
6. If the first of the remaining characters is in the Unicode Cyrillic character class, return them as-is.
7. Right-pad the remaining characters to 2 characters long using “a”s.
8. If either of the first two characters are not in the Unicode Letter character class, return “11”.
9. Return the characters as-is.

Examples

Testing

You can test Kobo’s prefix algorithm directly using dictword-test.

If you just want an easy way to generate prefixes for words, use the dictutil prefix command

Sample implementation

Here is the Go implementation used in dictutil:

func WordPrefix(word string) string {
	pfx := []rune(word)

	for i, c := range pfx {
		if i >= 2 || c == '\x00' { // limit to 2 chars, also cut at null
			pfx = pfx[:i] // trim up to current char
			break
		}
		pfx[i] = unicode.ToLower(c) // this includes accented chars
	}

	for len(pfx) != 0 {
		if unicode.IsSpace(pfx[0]) {
			pfx = pfx[1:] // trim left space
		} else {
			break
		}
	}

	for len(pfx) != 0 {
		if unicode.IsSpace(pfx[len(pfx)-1]) {
			pfx = pfx[:len(pfx)-1] // trim right space
		} else {
			break
		}
	}

	if len(pfx) == 0 {
		return "11" // if empty, return "11"
	}

	if !unicode.Is(unicode.Cyrillic, pfx[0]) {
		for len(pfx) < 2 {
			pfx = append(pfx, 'a') // pad right with 'a's to 2 chars
		}
		if !unicode.IsLetter(pfx[0]) || !unicode.IsLetter(pfx[1]) {
			return "11" // if either of the first 2 chars are letters, return "11"
		}
	}

	return string(pfx)
}