I'm wondering why we don't have some string classes that represent a string of Unicode grapheme clusters instead of code points or characters. It seems to me that in most applications it would be easier for programmers to access components of a grapheme when necessary than to have to organize them from code points, which appears necessary even if only to avoid casually breaking a string in "mid-grapheme" (at least in theory). Internally a string class might use a variable length encoding such as UTF-8, UTF-16, or in this context even UTF-32 is variable length; or implement subclasses for all of them (and optionally configure the choice at run-time so that different languages could use their optimal encodings). But if programmers could "see" grapheme units when inspecting a string, wouldn't string handling code in general be closer to achieving correctness, and without much extra complexity?

  • I guess a bit of time passed, and now we have a couple of languages which actually do this. :D – Trejkaz Nov 2 '17 at 0:09

Seems like the best way to get correctness is to keep programmers from doing "string hacking" ... it just isn't OK to write your own word wrap, hyphenation, word count, justification, cursor movement, etc. routines. All the modern UI frameworks will do this stuff for you these days.

That is, the abstraction you'd usually work with is more of a "paragraph display object," such as for GTK: http://library.gnome.org/devel/pango/stable/pango-Layout-Objects.html

rather than a grapheme string, such as: http://library.gnome.org/devel/pango/stable/pango-Glyph-Storage.html

To get to a string of glyphs you need info that's only available at the "view" level, so most uses of strings might not have this info. For example, you have to know the font, because fonts can have different ligatures.

Aside from that kind of practical matter, glyphs probably aren't what you want.

In many contexts, you want to use the proper Unicode attributes, shown in this API for example: http://library.gnome.org/devel/pango/stable/pango-Text-Processing.html#PangoLogAttr

As you can see from that struct (which reflects the Unicode algorithms) doing various things at glyph boundaries is not any more correct than doing them at character boundaries.

These two specs describe the algorithms to find different kinds of boundaries:

Doing text processing involves finding those boundaries with the algorithms and then working with the boundaries.

If you start digging in on just how hard it is to handle all languages correctly, you'll very quickly realize you need a library that looks at whole paragraphs and handles them properly. Windows, Mac, Linux (Qt and GTK), and Java all come with facilities for this, plus there's http://site.icu-project.org/ for example.

When writing web apps, unfortunately you pretty much have to let the browser (probably helped by the OS) do this stuff, as far as I know. All you can do in JavaScript or on the server side is mess it up.

Maybe I'd sum up the answer as: most string manipulation on natural language text is broken, so not much point worrying about the string class, other than maybe to have one with no methods on it ;-)

Your Answer

