Upgrade to Encode 2.14
[p5sagit/p5-mst-13.2.git] / ext / Encode / lib / Encode / Supported.pod
CommitLineData
5d030b67 1=head1 NAME
2
0ab8f81e 3Encode::Supported -- Encodings supported by Encode
5d030b67 4
5=head1 DESCRIPTION
6
5129552c 7=head2 Encoding Names
5d030b67 8
9Encoding names are case insensitive. White space in names
0ab8f81e 10is ignored. In addition, an encoding may have aliases.
5d030b67 11Each encoding has one "canonical" name. The "canonical"
12name is chosen from the names of the encoding by picking
a999c27c 13the first in the following sequence (with a few exceptions).
5d030b67 14
0ab8f81e 15=over 4
a999c27c 16
17=item *
18
962111ca 19The name used by the Perl community. That includes 'utf8' and 'ascii'.
20Unlike aliases, canonical names directly reach the method so such
21frequently used words like 'utf8' don't need to do alias lookups.
a999c27c 22
23=item *
24
0ab8f81e 25The MIME name as defined in IETF RFCs. This includes all "iso-"s.
a999c27c 26
27=item *
28
29The name in the IANA registry.
962111ca 30
a999c27c 31=item *
32
33The name used by the organization that defined it.
34
35=back
36
37In case I<de jure> canonical names differ from that of the Encode
38module, they are always aliased if it ever be implemented. So you can
39safely tell if a given encoding is implemented or not just by passing
40the canonical name.
5d030b67 41
5129552c 42Because of all the alias issues, and because in the general case
962111ca 43encodings have state, "Encode" uses an encoding object internally
5129552c 44once an operation is in progress.
5d030b67 45
5129552c 46=head1 Supported Encodings
5d030b67 47
48As of Perl 5.8.0, at least the following encodings are recognized.
49Note that unless otherwise specified, they are all case insensitive
962111ca 50(via alias) and all occurrence of spaces are replaced with '-'.
51In other words, "ISO 8859 1" and "iso-8859-1" are identical.
5d030b67 52
5129552c 53Encodings are categorized and implemented in several different modules
54but you don't have to C<use Encode::XX> to make them available for
962111ca 55most cases. Encode.pm will automatically load those modules on demand.
5d030b67 56
5129552c 57=head2 Built-in Encodings
5d030b67 58
5129552c 59The following encodings are always available.
5d030b67 60
962111ca 61 Canonical Aliases Comments & References
67d7b5ef 62 ----------------------------------------------------------------
2d06ad02 63 ascii US-ascii ISO-646-US [ECMA]
f0a41339 64 ascii-ctrl Special Encoding
962111ca 65 iso-8859-1 latin1 [ISO]
f0a41339 66 null Special Encoding
962111ca 67 utf8 UTF-8 [RFC2279]
c731e18e 68 ----------------------------------------------------------------
69
f0a41339 70I<null> and I<ascii-ctrl> are special. "null" fails for all character
71so when you set fallback mode to PERLQQ, HTMLCREF or XMLCREF, ALL
72CHARACTERS will fall back to character references. Ditto for
73"ascii-ctrl" except for control characters. For fallback modes, see
74L<Encode>.
75
c731e18e 76=head2 Encode::Unicode -- other Unicode encodings
77
78Unicode coding schemes other than native utf8 are supported by
0ab8f81e 79Encode::Unicode, which will be autoloaded on demand.
c731e18e 80
81 ----------------------------------------------------------------
f2a2953c 82 UCS-2BE UCS-2, iso-10646-1 [IANA, UC]
83 UCS-2LE [UC]
84 UTF-16 [UC]
85 UTF-16BE [UC]
86 UTF-16LE [UC]
87 UTF-32 [UC]
126bf8bf 88 UTF-32BE UCS-4 [UC]
f2a2953c 89 UTF-32LE [UC]
1485817e 90 UTF-7 [RFC2152]
67d7b5ef 91 ----------------------------------------------------------------
5d030b67 92
0ab8f81e 93To find how (UCS-2|UTF-(16|32))(LE|BE)? differ from one another,
f2a2953c 94see L<Encode::Unicode>.
95
1485817e 96UTF-7 is a special encoding which "re-encodes" UTF-16BE into a 7-bit
c2878c71 97encoding. It is implemented seperately by Encode::Unicode::UTF7.
1485817e 98
a999c27c 99=head2 Encode::Byte -- Extended ASCII
5d030b67 100
0ab8f81e 101Encode::Byte implements most single-byte encodings except for
102Symbols and EBCDIC. The following encodings are based on single-byte
103encodings implemented as extended ASCII. Most of them map
104\x80-\xff (upper half) to non-ASCII characters.
a999c27c 105
0ab8f81e 106=over 4
a999c27c 107
108=item ISO-8859 and corresponding vendor mappings
109
962111ca 110Since there are so many, they are presented in table format with
0ab8f81e 111languages and corresponding encoding names by vendors. Note that
112the table is sorted in order of ISO-8859 and the corresponding vendor
113mappings are slightly different from that of ISO. See
a999c27c 114L<http://czyborra.com/charsets/iso8859.html> for details.
115
962111ca 116 Lang/Regions ISO/Other Std. DOS Windows Macintosh Others
a999c27c 117 ----------------------------------------------------------------
962111ca 118 N. America (ASCII) cp437 AdobeStandardEncoding
119 cp863 (DOSCanadaF)
0ab8f81e 120 W. Europe iso-8859-1 cp850 cp1252 MacRoman nextstep
962111ca 121 hp-roman8
122 cp860 (DOSPortuguese)
123 Cntrl. Europe iso-8859-2 cp852 cp1250 MacCentralEurRoman
124 MacCroatian
125 MacRomanian
126 MacRumanian
ab3374e4 127 Latin3[1] iso-8859-3
128 Latin4[2] iso-8859-4
962111ca 129 Cyrillics iso-8859-5 cp855 cp1251 MacCyrillic
0ab8f81e 130 (See also next section) cp866 MacUkrainian
962111ca 131 Arabic iso-8859-6 cp864 cp1256 MacArabic
132 cp1006 MacFarsi
133 Greek iso-8859-7 cp737 cp1253 MacGreek
134 cp869 (DOSGreek2)
135 Hebrew iso-8859-8 cp862 cp1255 MacHebrew
136 Turkish iso-8859-9 cp857 cp1254 MacTurkish
137 Nordics iso-8859-10 cp865
138 cp861 MacIcelandic
139 MacSami
ab3374e4 140 Thai iso-8859-11[3] cp874 MacThai
a999c27c 141 (iso-8859-12 is nonexistent. Reserved for Indics?)
962111ca 142 Baltics iso-8859-13 cp775 cp1257
a999c27c 143 Celtics iso-8859-14
962111ca 144 Latin9 [4] iso-8859-15
a999c27c 145 Latin10 iso-8859-16
962111ca 146 Vietnamese viscii cp1258 MacVietnamese
a999c27c 147 ----------------------------------------------------------------
148
0ab8f81e 149 [1] Esperanto, Maltese, and Turkish. Turkish is now on 8859-9.
150 [2] Baltics. Now on 8859-10, except for Latvian.
ab3374e4 151 [3] TIS 620 + Non-Breaking Space (0xA0 / U+00A0)
0ab8f81e 152 [4] Nicknamed Latin0; the Euro sign as well as French and Finnish
153 letters that are missing from 8859-1 were added.
a999c27c 154
155All cp* are also available as ibm-*, ms-*, and windows-* . See also
156L<http://czyborra.com/charsets/codepages.html>.
157
158Macintosh encodings don't seem to be registered in such entities as
159IANA. "Canonical" names in Encode are based upon Apple's Tech Note
1601150. See L<http://developer.apple.com/technotes/tn/tn1150.html>
0ab8f81e 161for details.
a999c27c 162
0ab8f81e 163=item KOI8 - De Facto Standard for the Cyrillic world
a999c27c 164
0ab8f81e 165Though ISO-8859 does have ISO-8859-5, the KOI8 series is far more
166popular in the Net. L<Encode> comes with the following KOI charsets.
962111ca 167For gory details, see L<http://czyborra.com/charsets/cyrillic.html>
5d030b67 168
67d7b5ef 169 ----------------------------------------------------------------
962111ca 170 koi8-f
171 koi8-r cp878 [RFC1489]
172 koi8-u [RFC2319]
85982a32 173 ----------------------------------------------------------------
962111ca 174
a999c27c 175=item gsm0338 - Hentai Latin 1
176
962111ca 177GSM0338 is for GSM handsets. Though it shares alphanumerals with
178ASCII, control character ranges and other parts are mapped very
e74d7437 179differently, mainly to store Greek characters. There are also escape
180sequences (starting with 0x1B) to cover e.g. the Euro sign. Some
181special cases like a trailing 0x00 byte or a lone 0x1B byte are not
182well-defined and decode() will return an empty string for them.
183One possible workaround is
184
185 $gsm =~ s/\x00\z/\x00\x00/;
186 $uni = decode("gsm0338", $gsm);
187 $uni .= "\xA0" if $gsm =~ /\x1B\z/;
188
189Note that the Encode implementation of GSM0338 does not implement the
190reuse of Latin capital letters as Greek capital letters (for example,
191the 0x5A is U+005A (LATIN CAPITAL LETTER Z), not U+0396 (GREEK CAPITAL
192LETTER ZETA).
193
194The GSM0338 is also covered in Encode::Byte even though it is not
195an "extended ASCII" encoding.
a999c27c 196
197=back
5d030b67 198
0ab8f81e 199=head2 CJK: Chinese, Japanese, Korean (Multibyte)
5d030b67 200
962111ca 201Note that Vietnamese is listed above. Also read "Encoding vs Charset"
0ab8f81e 202below. Also note that these are implemented in distinct modules by
ab3374e4 203countries, due to the size concerns (simplified Chinese is mapped
0ab8f81e 204to 'CN', continental China, while traditional Chinese is mapped to
ab3374e4 205'TW', Taiwan). Please refer to their respective documentation pages.
5d030b67 206
5129552c 207=over 4
208
209=item Encode::CN -- Continental China
210
962111ca 211 Standard DOS/Win Macintosh Comment/Reference
67d7b5ef 212 ----------------------------------------------------------------
962111ca 213 euc-cn [1] MacChineseSimp
214 (gbk) cp936 [2]
215 gb12345-raw { GB12345 without CES }
216 gb2312-raw { GB2312 without CES }
5129552c 217 hz
218 iso-ir-165
67d7b5ef 219 ----------------------------------------------------------------
5129552c 220
0ab8f81e 221 [1] GB2312 is aliased to this. See L<Microsoft-related naming mess>
222 [2] gbk is aliased to this. See L<Microsoft-related naming mess>
f2a2953c 223
5129552c 224=item Encode::JP -- Japan
225
962111ca 226 Standard DOS/Win Macintosh Comment/Reference
67d7b5ef 227 ----------------------------------------------------------------
a999c27c 228 euc-jp
962111ca 229 shiftjis cp932 macJapanese
f2a2953c 230 7bit-jis
962111ca 231 iso-2022-jp [RFC1468]
232 iso-2022-jp-1 [RFC2237]
f2a2953c 233 jis0201-raw { JIS X 0201 (roman + halfwidth kana) without CES }
234 jis0208-raw { JIS X 0208 (Kanji + fullwidth kana) without CES }
235 jis0212-raw { JIS X 0212 (Extended Kanji) without CES }
67d7b5ef 236 ----------------------------------------------------------------
5129552c 237
238=item Encode::KR -- Korea
239
962111ca 240 Standard DOS/Win Macintosh Comment/Reference
67d7b5ef 241 ----------------------------------------------------------------
962111ca 242 euc-kr MacKorean [RFC1557]
243 cp949 [1]
244 iso-2022-kr [RFC1557]
a999c27c 245 johab [KS X 1001:1998, Annex 3]
f2a2953c 246 ksc5601-raw { KSC5601 without CES }
67d7b5ef 247 ----------------------------------------------------------------
5129552c 248
962111ca 249 [1] ks_c_5601-1987, (x-)?windows-949, and uhc are aliased to this.
250 See below.
251
5129552c 252=item Encode::TW -- Taiwan
253
962111ca 254 Standard DOS/Win Macintosh Comment/Reference
67d7b5ef 255 ----------------------------------------------------------------
b0b300a3 256 big5-eten cp950 MacChineseTrad {big5 aliased to big5-eten}
257 big5-hkscs
67d7b5ef 258 ----------------------------------------------------------------
5129552c 259
260=item Encode::HanExtra -- More Chinese via CPAN
261
ab3374e4 262Due to the size concerns, additional Chinese encodings below are
5129552c 263distributed separately on CPAN, under the name Encode::HanExtra.
264
962111ca 265 Standard DOS/Win Macintosh Comment/Reference
67d7b5ef 266 ----------------------------------------------------------------
e8c86ba6 267 big5ext CMEX's Big5e Extension
268 big5plus CMEX's Big5+ Extension
269 cccii Chinese Character Code for Information Interchange
270 euc-tw EUC (Extended Unix Character)
271 gb18030 GBK with Traditional Characters
272 ----------------------------------------------------------------
273
274=item Encode::JIS2K -- JIS X 0213 encodings via CPAN
275
276Due to size concerns, additional Japanese encodings below are
277distributed separately on CPAN, under the name Encode::JIS2K.
278
279 Standard DOS/Win Macintosh Comment/Reference
280 ----------------------------------------------------------------
281 euc-jisx0213
282 shiftjisx0123
283 iso-2022-jp-3
284 jis0213-1-raw
285 jis0213-2-raw
67d7b5ef 286 ----------------------------------------------------------------
5129552c 287
288=back
289
290=head2 Miscellaneous encodings
291
292=over 4
293
294=item Encode::EBCDIC
5d030b67 295
a999c27c 296See L<perlebcdic> for details.
5d030b67 297
67d7b5ef 298 ----------------------------------------------------------------
5d030b67 299 cp37
a999c27c 300 cp500
301 cp875
302 cp1026
303 cp1047
5d030b67 304 posix-bc
67d7b5ef 305 ----------------------------------------------------------------
5129552c 306
a63c962f 307=item Encode::Symbols
5d030b67 308
5129552c 309For symbols and dingbats.
5d030b67 310
67d7b5ef 311 ----------------------------------------------------------------
5d030b67 312 symbol
313 dingbats
a999c27c 314 MacDingbats
315 AdobeZdingbat
316 AdobeSymbol
67d7b5ef 317 ----------------------------------------------------------------
318
e8c86ba6 319=item Encode::MIME::Header
320
321Strictly speaking, MIME header encoding documented in RFC 2047 is more
ab3374e4 322of encapsulation than encoding. However, their support in modern
323world is imperative so they are supported.
e8c86ba6 324
325 ----------------------------------------------------------------
326 MIME-Header [RFC2047]
327 MIME-B [RFC2047]
328 MIME-Q [RFC2047]
329 ----------------------------------------------------------------
330
331=item Encode::Guess
332
333This one is not a name of encoding but a utility that lets you pick up
334the most appropriate encoding for a data out of given I<suspects>. See
335L<Encode::Guess> for details.
336
67d7b5ef 337=back
338
339=head1 Unsupported encodings
340
0ab8f81e 341The following encodings are not supported as yet; some because they
342are rarely used, some because of technical difficulties. They may
343be supported by external modules via CPAN in the future, however.
67d7b5ef 344
345=over 4
346
347=item ISO-2022-JP-2 [RFC1554]
348
349Not very popular yet. Needs Unicode Database or equivalent to
0ab8f81e 350implement encode() (because it includes JIS X 0208/0212, KSC5601, and
351GB2312 simultaneously, whose code points in Unicode overlap. So you
352need to lookup the database to determine to what character set a given
67d7b5ef 353Unicode character should belong).
354
962111ca 355=item ISO-2022-CN [RFC1922]
67d7b5ef 356
0ab8f81e 357Not very popular. Needs CNS 11643-1 and -2 which are not available in
962111ca 358this module. CNS 11643 is supported (via euc-tw) in Encode::HanExtra.
0ab8f81e 359Autrijus Tang may add support for this encoding in his module in future.
67d7b5ef 360
0ab8f81e 361=item Various HP-UX encodings
67d7b5ef 362
962111ca 363The following are unsupported due to the lack of mapping data.
364
67d7b5ef 365 '8' - arabic8, greek8, hebrew8, kana8, thai8, and turkish8
962111ca 366 '15' - japanese15, korean15, and roi15
67d7b5ef 367
368=item Cyrillic encoding ISO-IR-111
369
0ab8f81e 370Anton Tagunov doubts its usefulness.
67d7b5ef 371
372=item ISO-8859-8-1 [Hebrew]
373
a999c27c 374None of the Encode team knows Hebrew enough (ISO-8859-8, cp1255 and
375MacHebrew are supported because and just because there were mappings
962111ca 376available at L<http://www.unicode.org/>). Contributions welcome.
377
378=item ISIRI 3342, Iran System, ISIRI 2900 [Farsi]
379
380Ditto.
67d7b5ef 381
382=item Thai encoding TCVN
383
384Ditto.
385
386=item Vietnamese encodings VPS
387
0ab8f81e 388Though Jungshik Shin has reported that Mozilla supports this encoding,
389it was too late before 5.8.0 for us to add it. In the future, it
390may be available via a separate module. See
962111ca 391L<http://lxr.mozilla.org/seamonkey/source/intl/uconv/ucvlatin/vps.uf>
392and
a999c27c 393L<http://lxr.mozilla.org/seamonkey/source/intl/uconv/ucvlatin/vps.ut>
394if you are interested in helping us.
67d7b5ef 395
962111ca 396=item Various Mac encodings
67d7b5ef 397
962111ca 398The following are unsupported due to the lack of mapping data.
a999c27c 399
400 MacArmenian, MacBengali, MacBurmese, MacEthiopic
401 MacExtArabic, MacGeorgian, MacKannada, MacKhmer
402 MacLaotian, MacMalayalam, MacMongolian, MacOriya
403 MacSinhalese, MacTamil, MacTelugu, MacTibetan
404 MacVietnamese
405
0ab8f81e 406The rest which are already available are based upon the vendor mappings
962111ca 407at L<http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/> .
a999c27c 408
409=item (Mac) Indic encodings
410
0ab8f81e 411The maps for the following are available at L<http://www.unicode.org/>
412but remain unsupport because those encodings need algorithmical
413approach, currently unsupported by F<enc2xs>:
67d7b5ef 414
a999c27c 415 MacDevanagari
416 MacGurmukhi
417 MacGujarati
67d7b5ef 418
a999c27c 419For details, please see C<Unicode mapping issues and notes:> at
420L<http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/DEVANAGA.TXT> .
421
422I believe this issue is prevalent not only for Mac Indics but also in
962111ca 423other Indic encodings, but the above were the only Indic encodings
a999c27c 424maps that I could find at L<http://www.unicode.org/> .
5129552c 425
426=back
5d030b67 427
a999c27c 428=head1 Encoding vs. Charset -- terminology
5d030b67 429
0ab8f81e 430We are used to using the term (character) I<encoding> and I<character
431set> interchangeably. But just as confusing the terms byte and
432character is dangerous and the terms should be differentiated when
433needed, we need to differentiate I<encoding> and I<character set>.
5d030b67 434
0ab8f81e 435To understand that, here is a description of how we make computers
436grok our characters.
a999c27c 437
438=over 4
439
440=item *
67d7b5ef 441
a999c27c 442First we start with which characters to include. We call this
443collection of characters I<character repertoire>.
5d030b67 444
a999c27c 445=item *
5d030b67 446
a999c27c 447Then we have to give each character a unique ID so your computer can
0ab8f81e 448tell the difference between 'a' and 'A'. This itemized character
962111ca 449repertoire is now a I<character set>.
a63c962f 450
a999c27c 451=item *
452
453If your computer can grow the character set without further
0ab8f81e 454processing, you can go ahead and use it. This is called a I<coded
a999c27c 455character set> (CCS) or I<raw character encoding>. ASCII is used this
456way for most cases.
457
458=item *
459
0ab8f81e 460But in many cases, especially multi-byte CJK encodings, you have to
a999c27c 461tweak a little more. Your network connection may not accept any data
0ab8f81e 462with the Most Significant Bit set, and your computer may not be able to
a999c27c 463tell if a given byte is a whole character or just half of it. So you
464have to I<encode> the character set to use it.
465
466A I<character encoding scheme> (CES) determines how to encode a given
467character set, or a set of multiple character sets. 7bit ISO-2022 is
0ab8f81e 468an example of a CES. You switch between character sets via I<escape
469sequences>.
67d7b5ef 470
471=back
472
0ab8f81e 473Technically, or mathematically, speaking, a character set encoded in
a999c27c 474such a CES that maps character by character may form a CCS. EUC is such
0ab8f81e 475an example. The CES of EUC is as follows:
67d7b5ef 476
a999c27c 477=over 4
5d030b67 478
a999c27c 479=item *
5d030b67 480
a999c27c 481Map ASCII unchanged.
482
483=item *
484
485Map such a character set that consists of 94 or 96 powered by N
486members by adding 0x80 to each byte.
487
488=item *
489
0ab8f81e 490You can also use 0x8e and 0x8f to indicate that the following sequence of
491characters belongs to yet another character set. To each following byte
492is added the value 0x80.
a999c27c 493
494=back
495
0ab8f81e 496By carefully looking at the encoded byte sequence, you can find that the
497byte sequence conforms a unique number. In that sense, EUC is a CCS
a999c27c 498generated by a CES above from up to four CCS (complicated?). UTF-8
0ab8f81e 499falls into this category. See L<perlUnicode/"UTF-8"> to find out how
a999c27c 500UTF-8 maps Unicode to a byte sequence.
501
0ab8f81e 502You may also have found out by now why 7bit ISO-2022 cannot comprise
503a CCS. If you look at a byte sequence \x21\x21, you can't tell if
504it is two !'s or IDEOGRAPHIC SPACE. EUC maps the latter to \xA1\xA1
505so you have no trouble differentiating between "!!". and S<" ">.
67d7b5ef 506
a63c962f 507=head1 Encoding Classification (by Anton Tagunov and Dan Kogai)
508
509This section tries to classify the supported encodings by their
510applicability for information exchange over the Internet and to
511choose the most suitable aliases to name them in the context of
512such communication.
513
0ab8f81e 514=over 4
67d7b5ef 515
516=item *
517
0ab8f81e 518To (en|de)code encodings marked by C<(**)>, you need
a999c27c 519C<Encode::HanExtra>, available from CPAN.
67d7b5ef 520
521=back
522
a63c962f 523Encoding names
5d030b67 524
f2a2953c 525 US-ASCII UTF-8 ISO-8859-* KOI8-R
526 Shift_JIS EUC-JP ISO-2022-JP ISO-2022-JP-1
527 EUC-KR Big5 GB2312
a999c27c 528
0ab8f81e 529are registered with IANA as preferred MIME names and may
a999c27c 530be used over the Internet.
5d030b67 531
c731e18e 532C<Shift_JIS> has been officialized by JIS X 0208:1997.
a999c27c 533L<Microsoft-related naming mess> gives details.
5d030b67 534
a999c27c 535C<GB2312> is the IANA name for C<EUC-CN>.
536See L<Microsoft-related naming mess> for details.
537
538C<GB_2312-80> I<raw> encoding is available as C<gb2312-raw>
f2a2953c 539with Encode. See L<Encode::CN> for details.
5d030b67 540
a63c962f 541 EUC-CN
f2a2953c 542 KOI8-U [RFC2319]
5d030b67 543
a999c27c 544have not been registered with IANA (as of March 2002) but
545seem to be supported by major web browsers.
0ab8f81e 546The IANA name for C<EUC-CN> is C<GB2312>.
67d7b5ef 547
548 KS_C_5601-1987
549
a999c27c 550is heavily misused.
551See L<Microsoft-related naming mess> for details.
552
553C<KS_C_5601-1987> I<raw> encoding is available as C<kcs5601-raw>
f2a2953c 554with Encode. See L<Encode::KR> for details.
555
556 UTF-16 UTF-16BE UTF-16LE
557
448e90bb 558are IANA-registered C<charset>s. See [RFC 2781] for details.
f2a2953c 559Jungshik Shin reports that UTF-16 with a BOM is well accepted
560by MS IE 5/6 and NS 4/6. Beware however that
561
0ab8f81e 562=over 4
f2a2953c 563
564=item *
5d030b67 565
f2a2953c 566C<UTF-16> support in any software you're going to be
567using/interoperating with has probably been less tested
568then C<UTF-8> support
5d030b67 569
f2a2953c 570=item *
571
c731e18e 572C<UTF-8> coded data seamlessly passes traditional
573command piping (C<cat>, C<more>, etc.) while C<UTF-16> coded
0ab8f81e 574data is likely to cause confusion (with its zero bytes,
f2a2953c 575for example)
576
577=item *
578
579it is beyond the power of words to describe the way HTML browsers
0ab8f81e 580encode non-C<ASCII> form data. To get a general impression, visit
f2a2953c 581L<http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html>.
0ab8f81e 582While encoding of form data has stabilized for C<UTF-8> encoded pages
583(at least IE 5/6, NS 6, and Opera 6 behave consistently), be sure to
584expect fun (and cross-browser discrepancies) with C<UTF-16> encoded
f2a2953c 585pages!
586
587=back
588
589The rule of thumb is to use C<UTF-8> unless you know what
c731e18e 590you're doing and unless you really benefit from using C<UTF-16>.
a999c27c 591
f2a2953c 592 ISO-IR-165 [RFC1345]
5d030b67 593 VISCII
a63c962f 594 GB 12345
f2a2953c 595 GB 18030 (**) (see links bellow)
596 EUC-TW (**)
5d030b67 597
598are totally valid encodings but not registered at IANA.
a63c962f 599The names under which they are listed here are probably the
600most widely-known names for these encodings and are recommended
601names.
602
f2a2953c 603 BIG5PLUS (**)
a63c962f 604
0ab8f81e 605is a proprietary name.
5d030b67 606
a999c27c 607=head2 Microsoft-related naming mess
608
609Microsoft products misuse the following names:
5d030b67 610
0ab8f81e 611=over 4
a63c962f 612
a999c27c 613=item KS_C_5601-1987
5d030b67 614
a999c27c 615Microsoft extension to C<EUC-KR>.
5d030b67 616
c731e18e 617Proper names: C<CP949>, C<UHC>, C<x-windows-949> (as used by Mozilla).
67d7b5ef 618
f2a2953c 619See L<http://lists.w3.org/Archives/Public/ietf-charsets/2001AprJun/0033.html>
a999c27c 620for details.
5d030b67 621
f2a2953c 622Encode aliases C<KS_C_5601-1987> to C<cp949> to reflect this common
623misusage. I<Raw> C<KS_C_5601-1987> encoding is available as
624C<kcs5601-raw>.
5d030b67 625
f2a2953c 626See L<Encode::KR> for details.
67d7b5ef 627
a999c27c 628=item GB2312
67d7b5ef 629
a999c27c 630Microsoft extension to C<EUC-CN>.
a63c962f 631
a999c27c 632Proper names: C<CP936>, C<GBK>.
a63c962f 633
a999c27c 634C<GB2312> has been registered in the C<EUC-CN> meaning at
635IANA. This has partially repaired the situation: Microsoft's
636C<GB2312> has become a superset of the official C<GB2312>.
67d7b5ef 637
a999c27c 638Encode aliases C<GB2312> to C<euc-cn> in full agreement with
639IANA registration. C<cp936> is supported separately.
f2a2953c 640I<Raw> C<GB_2312-80> encoding is available as C<gb2312-raw>.
a999c27c 641
f2a2953c 642See L<Encode::CN> for details.
a999c27c 643
644=item Big5
645
646Microsoft extension to C<Big5>.
647
648Proper name: C<CP950>.
649
650Encode separately supports C<Big5> and C<cp950>.
651
652=item Shift_JIS
653
654Microsoft's understanding of C<Shift_JIS>.
655
656JIS has not endorsed the full Microsoft standard however.
657The official C<Shift_JIS> includes only JIS X 0201 and JIS X 0208
0ab8f81e 658character sets, while Microsoft has always used C<Shift_JIS>
85982a32 659to encode a wider character repertoire. See C<IANA> registration for
c731e18e 660C<Windows-31J>.
a999c27c 661
0ab8f81e 662As a historical predecessor, Microsoft's variant
663probably has more rights for the name, though it may be objected
a999c27c 664that Microsoft shouldn't have used JIS as part of the name
665in the first place.
666
8f1ed24a 667Unambiguous name: C<CP932>. C<IANA> name (also used by Mozilla, and
668provided as an alias by Encode): C<Windows-31J>.
a999c27c 669
670Encode separately supports C<Shift_JIS> and C<cp932>.
671
672=back
673
674=head1 Glossary
675
0ab8f81e 676=over 4
a999c27c 677
678=item character repertoire
679
0ab8f81e 680A collection of unique characters. A I<character> set in the strictest
681sense. At this stage, characters are not numbered.
a999c27c 682
683=item coded character set (CCS)
684
685A character set that is mapped in a way computers can use directly.
0ab8f81e 686Many character encodings, including EUC, fall in this category.
a999c27c 687
688=item character encoding scheme (CES)
689
690An algorithm to map a character set to a byte sequence. You don't
691have to be able to tell which character set a given byte sequence
692belongs. 7-bit ISO-2022 is a CES but it cannot be a CCS. EUC is an
693example of being both a CCS and CES.
694
f2a2953c 695=item charset (in MIME context)
696
697has long been used in the meaning of C<encoding>, CES.
698
0ab8f81e 699While the word combination C<character set> has lost this meaning
700in MIME context since [RFC 2130], the C<charset> abbreviation has
701retained it. This is how [RFC 2277] and [RFC 2278] bless C<charset>:
f2a2953c 702
703 This document uses the term "charset" to mean a set of rules for
704 mapping from a sequence of octets to a sequence of characters, such
705 as the combination of a coded character set and a character encoding
706 scheme; this is also what is used as an identifier in MIME "charset="
707 parameters, and registered in the IANA charset registry ... (Note
708 that this is NOT a term used by other standards bodies, such as ISO).
ab3374e4 709 [RFC 2277]
f2a2953c 710
a999c27c 711=item EUC
712
0ab8f81e 713Extended Unix Character. See ISO-2022.
a999c27c 714
715=item ISO-2022
716
0ab8f81e 717A CES that was carefully designed to coexist with ASCII. There are a 7
718bit version and an 8 bit version.
f2a2953c 719
0ab8f81e 720The 7 bit version switches character set via escape sequence so it
f2a2953c 721cannot form a CCS. Since this is more difficult to handle in programs
0ab8f81e 722than the 8 bit version, the 7 bit version is not very popular except for
723iso-2022-jp, the I<de facto> standard CES for e-mails.
f2a2953c 724
0ab8f81e 725The 8 bit version can form a CCS. EUC and ISO-8859 are two examples
962111ca 726thereof. Pre-5.6 perl could use them as string literals.
a999c27c 727
728=item UCS
729
730Short for I<Universal Character Set>. When you say just UCS, it means
0ab8f81e 731I<Unicode>.
a999c27c 732
733=item UCS-2
734
735ISO/IEC 10646 encoding form: Universal Character Set coded in two
736octets.
737
738=item Unicode
739
0ab8f81e 740A character set that aims to include all character repertoires of the
962111ca 741world. Many character sets in various national as well as industrial
f2a2953c 742standards have become, in a way, just subsets of Unicode.
a999c27c 743
744=item UTF
745
f2a2953c 746Short for I<Unicode Transformation Format>. Determines how to map a
0ab8f81e 747Unicode character into a byte sequence.
a999c27c 748
749=item UTF-16
750
751A UTF in 16-bit encoding. Can either be in big endian or little
0ab8f81e 752endian. The big endian version is called UTF-16BE (equal to UCS-2 +
753surrogate support) and the little endian version is called UTF-16LE.
67d7b5ef 754
755=back
5d030b67 756
757=head1 See Also
758
5129552c 759L<Encode>,
760L<Encode::Byte>,
a63c962f 761L<Encode::CN>, L<Encode::JP>, L<Encode::KR>, L<Encode::TW>,
5129552c 762L<Encode::EBCDIC>, L<Encode::Symbol>
e8c86ba6 763L<Encode::MIME::Header>, L<Encode::Guess>
5d030b67 764
a999c27c 765=head1 References
766
0ab8f81e 767=over 4
a999c27c 768
769=item ECMA
770
771European Computer Manufacturers Association
772L<http://www.ecma.ch>
773
0ab8f81e 774=over 4
a999c27c 775
0ab8f81e 776=item ECMA-035 (eq C<ISO-2022>)
a999c27c 777
778L<http://www.ecma.ch/ecma1/STAND/ECMA-035.HTM>
779
0ab8f81e 780The specification of ISO-2022 is available from the link above.
a999c27c 781
782=back
783
784=item IANA
785
786Internet Assigned Numbers Authority
787L<http://www.iana.org/>
788
0ab8f81e 789=over 4
a999c27c 790
791=item Assigned Charset Names by IANA
792
793L<http://www.iana.org/assignments/character-sets>
794
795Most of the C<canonical names> in Encode derive from this list
796so you can directly apply the string you have extracted from MIME
448e90bb 797header of mails and web pages.
a999c27c 798
799=back
800
801=item ISO
802
803International Organization for Standardization
804L<http://www.iso.ch/>
805
806=item RFC
807
962111ca 808Request For Comments -- need I say more?
0ab8f81e 809L<http://www.rfc-editor.org/>, L<http://www.rfc.net/>,
810L<http://www.faqs.org/rfcs/>
a999c27c 811
812=item UC
813
814Unicode Consortium
815L<http://www.unicode.org/>
816
0ab8f81e 817=over 4
a999c27c 818
819=item Unicode Glossary
820
821L<http://www.unicode.org/glossary/>
822
962111ca 823The glossary of this document is based upon this site.
a999c27c 824
825=back
826
827=back
828
829=head2 Other Notable Sites
830
0ab8f81e 831=over 4
a999c27c 832
833=item czyborra.com
834
f2a2953c 835L<http://czyborra.com/>
a999c27c 836
cf525c36 837Contains a lot of useful information, especially gory details of ISO
a999c27c 838vs. vendor mappings.
839
840=item CJK.inf
841
842L<http://www.oreilly.com/people/authors/lunde/cjk_inf.html>
843
844Somewhat obsolete (last update in 1996), but still useful. Also try
845
846L<ftp://ftp.oreilly.com/pub/examples/nutshell/cjkv/pdf/GB18030_Summary.pdf>
847
0ab8f81e 848You will find brief info on C<EUC-CN>, C<GBK> and mostly on C<GB 18030>.
a999c27c 849
f2a2953c 850=item Jungshik Shin's Hangul FAQ
851
852L<http://jshin.net/faq>
853
0ab8f81e 854And especially its subject 8.
f2a2953c 855
856L<http://jshin.net/faq/qa8.html>
857
962111ca 858A comprehensive overview of the Korean (C<KS *>) standards.
f2a2953c 859
0ab8f81e 860=item debian.org: "Introduction to i18n"
861
862A brief description for most of the mentioned CJK encodings is
863contained in
864L<http://www.debian.org/doc/manuals/intro-i18n/ch-codes.en.html>
865
f2a2953c 866=back
867
868=head2 Offline sources
869
0ab8f81e 870=over 4
f2a2953c 871
872=item C<CJKV Information Processing> by Ken Lunde
873
874CJKV Information Processing
8751999 O'Reilly & Associates, ISBN : 1-56592-224-7
876
0ab8f81e 877The modern successor of C<CJK.inf>.
f2a2953c 878
0ab8f81e 879Features a comprehensive coverage of CJKV character sets and
f2a2953c 880encodings along with many other issues faced by anyone trying
881to better support CJKV languages/scripts in all the areas of
882information processing.
883
0ab8f81e 884To purchase this book, visit
f2a2953c 885L<http://www.oreilly.com/catalog/cjkvinfo/>
0ab8f81e 886or your favourite bookstore.
f2a2953c 887
a999c27c 888=back
889
5d030b67 890=cut