Section: Linux Programmer's Manual (7)
Updated: 2008-06-03
Index Return to Main Contents


charsets - programmer's view of character sets and internationalization  


Linux is an international operating system. Various of its utilities and device drivers (including the console driver) support multilingual character sets including Latin-alphabet letters with diacritical marks, accents, ligatures, and entire non-Latin alphabets including Greek, Cyrillic, Arabic, and Hebrew.

This manual page presents a programmer's-eye view of different character-set standards and how they fit together on Linux. Standards discussed include ASCII, ISO 8859, KOI8-R, Unicode, ISO 2022 and ISO 4873. The primary emphasis is on character sets actually used as locale character sets, not the myriad others that can be found in data from other systems.

A complete list of charsets used in an officially supported locale in glibc 2.2.3 is: ISO-8859-{1,2,3,5,6,7,8,9,13,15}, CP1251, UTF-8, EUC-{KR,JP,TW}, KOI8-{R,U}, GB2312, GB18030, GBK, BIG5, BIG5-HKSCS and TIS-620 (in no particular order.) (Romanian may be switching to ISO-8859-16.)  


ASCII (American Standard Code For Information Interchange) is the original 7-bit character set, originally designed for American English. It is currently described by the ECMA-6 standard.

Various ASCII variants replacing the dollar sign with other currency symbols and replacing punctuation with non-English alphabetic characters to cover German, French, Spanish and others in 7 bits exist. All are deprecated; glibc doesn't support locales whose character sets aren't true supersets of ASCII. (These sets are also known as ISO-646, a close relative of ASCII that permitted replacing these characters.)

As Linux was written for hardware designed in the US, it natively supports ASCII.  

ISO 8859

ISO 8859 is a series of 15 8-bit character sets all of which have US ASCII in their low (7-bit) half, invisible control characters in positions 128 to 159, and 96 fixed-width graphics in positions 160-255.

Of these, the most important is ISO 8859-1 (Latin-1). It is natively supported in the Linux console driver, fairly well supported in X11R6, and is the base character set of HTML.


Here are brief descriptions of each set:

8859-1 (Latin-1)
Latin-1 covers most Western European languages such as Albanian, Catalan, Danish, Dutch, English, Faroese, Finnish, French, German, Galician, Irish, Icelandic, Italian, Norwegian, Portuguese, Spanish, and Swedish. The lack of the ligatures Dutch ij, French oe and old-style ,,German`` quotation marks is considered tolerable.
8859-2 (Latin-2)
Latin-2 supports most Latin-written Slavic and Central European languages: Croatian, Czech, German, Hungarian, Polish, Rumanian, Slovak, and Slovene.
8859-3 (Latin-3)
Latin-3 is popular with authors of Esperanto, Galician, and Maltese. (Turkish is now written with 8859-9 instead.)
8859-4 (Latin-4)
Latin-4 introduced letters for Estonian, Latvian, and Lithuanian. It is essentially obsolete; see 8859-10 (Latin-6) and 8859-13 (Latin-7).
Cyrillic letters supporting Bulgarian, Byelorussian, Macedonian, Russian, Serbian and Ukrainian. Ukrainians read the letter "ghe" with downstroke as "heh" and would need a ghe with upstroke to write a correct ghe. See the discussion of KOI8-R below.
Supports Modern Greek.
Supports modern Hebrew without niqud (punctuation signs). Niqud and full-fledged Biblical Hebrew are outside the scope of this character set; under Linux, UTF-8 is the preferred encoding for these.
8859-9 (Latin-5)
8859-10 (Latin-6)
Latin 6 adds the last Inuit (Greenlandic) and Sami (Lappish) letters that were missing in Latin 4 to cover the entire Nordic area. RFC 1345 listed a preliminary and different "latin6". Skolt Sami still needs a few more accents than these.
This only exists as a rejected draft standard. The draft standard was identical to TIS-620, which is used under Linux for Thai.
This set does not exist. While Vietnamese has been suggested for this space, it does not fit within the 96 (non-combining) characters ISO 8859 offers. UTF-8 is the preferred character set for Vietnamese use under Linux.
8859-13 (Latin-7)
8859-14 (Latin-8)
This is the Celtic character set, covering Gaelic and Welsh. This charset also contains the dotted characters needed for Old Irish.
8859-15 (Latin-9)
This adds the Euro sign and French and Finnish letters that were missing in Latin-1.
8859-16 (Latin-10)
This set covers many of the languages covered by 8859-2, and supports Romanian more completely then that set does.


KOI8-R is a non-ISO character set popular in Russia. The lower half is US ASCII; the upper is a Cyrillic character set somewhat better designed than ISO 8859-5. KOI8-U is a common character set, based off KOI8-R, that has better support for Ukrainian. Neither of these sets are ISO-2022 compatible, unlike the ISO-8859 series.


JIS X 0208


KS X 1001


GB 2312



Big5 is a popular character set in Taiwan to express traditional Chinese. (Big5 is both a character set and an encoding.) It is a superset of US ASCII. Non-ASCII characters are expressed in two bytes. Bytes 0xa1-0xfe are used as leading bytes for two-byte characters. Big5 and its extension is widely used in Taiwan and Hong Kong. It is not ISO 2022-compliant.  

TIS 620

TIS 620 is a Thai national standard character set and a superset of US ASCII. Like ISO 8859 series, Thai characters are mapped into 0xa1-0xfe. TIS 620 is the only commonly used character set under Linux besides UTF-8 to have combining characters.  






For most people who use ISO-8859 character sets, this means that the characters outside of ASCII are now coded with two bytes. This tends to expand ordinary text files by only one or two percent. For Russian or Greek users, this expands ordinary text files by 100%, since text in those languages is mostly outside of ASCII. For Japanese users this means that the 16-bit codes now in common use will take three bytes. While there are algorithmic conversions from some character sets (esp. ISO-8859-1) to Unicode, general conversion requires carrying around conversion tables, which can be quite large for 16-bit codes.

Note that UTF-8 is self-synchronizing: 10xxxxxx is a tail, any other byte is the head of a code. Note that the only way ASCII bytes occur in a UTF-8 stream, is as themselves. In particular, there are no embedded NULs (aq\0aq) or aq/aqs that form part of some larger code.

Since ASCII, and, in particular, NUL and aq/aq, are unchanged, the kernel does not notice that UTF-8 is being used. It does not care at all what the bytes it is handling stand for.

Rendering of Unicode data streams is typically handled through "subfont" tables which map a subset of Unicode to glyphs. Internally the kernel uses Unicode to describe the subfont loaded in video RAM. This means that in UTF-8 mode one can use a character set with 512 different symbols. This is not enough for Japanese, Chinese and Korean, but it is enough for most other purposes.

At the current time, the console driver does not handle combining characters. So Thai, Sioux and any other script needing combining characters can't be handled on the console.  

ISO 2022 and ISO 4873














ISO 8859
JIS X 0208
KS X 1001
GB 2312
TIS 620
ISO 2022 and ISO 4873

This document was created by man2html, using the manual pages.
Time: 05:34:14 GMT, December 24, 2015