Question

I m trying to evaluate different strategies for case insensitive UTF-8 string comparison.

I ve read some material from the Unicode consortium, experimented with ICU and tried to come up with various quality-of-implementation alternatives.

On multiple occasions I ve seen texts differ between Simple Case Mapping and Full Case Mapping, and I wanted to make sure I understand the difference entirely.

As I read it, Simple Case Mapping is "context-free", i.e. doesn t need to know what language the payload is. This will give approximate results, due to the Turkic "I/ı/İ/i" debacle.

Full Case Mapping, on the other hand, needs to know the language of the payload to be able to perform the mapping. With that extra information, it can take special measures to cover cases where "Kim" as a Turkic string should become "KİM" in upper-case, but "Kim" as an English string, should become "KIM" in upper-case.

Have I got that right?

Are there other examples of "multi-faceted" code points that fold differently for different languages?

Thanks!

UPDATE: One of the sources mentioning simple case mapping as language independent is ICU s documentation. I interpreted that as Unicode truth, but maybe it s just a statement of the implementation?

Answer 1

No, a "full case mapping" is a casing where one codepoint needs to be replaced by more than one new codepoints. A simple case mapping is a single codepoint substitution.

If you want to implement this yourself then the Unicode CaseFolding.txt file is crucial to get this right. Note the status field code "T", specifically there to handle the Turkish I problem.

Answer 2

Well ... The consonant combination "SS" would down-case to "ss" for most Western languages, but in German it might become the special letter "ß". That s just "might", there are quite involved usage rules to consider.

I think this doesn t directly affect collation order (any Germans are of course welcome to correct me) though, so maybe it s a moot point.

友情链接