=head1 NAME
-Encode::Supports -- Supported encodings by Encode
+Encode::Supported -- Supported encodings by Encode
=head1 DESCRIPTION
is ignored. In addition an encoding may have aliases.
Each encoding has one "canonical" name. The "canonical"
name is chosen from the names of the encoding by picking
-he first in the following sequence:
+the first in the following sequence (with a few exceptions).
- o The MIME name as defined in IETF RFCs.
- o The name in the IANA registry.
- o The name used by the organization that defined it.
+=over
+
+=item *
+
+The name used by the perl community. That includes 'utf8' and 'ascii'.
+Unlike aliases, canonical names directly reaches the method so such
+frequently used words like 'utf8' should do without alias lookups.
+
+=item *
+
+The MIME name as defined in IETF RFCs This includes all "iso-"'s.
+
+=item *
+
+The name in the IANA registry.
+
+=item *
+
+The name used by the organization that defined it.
+
+=back
+
+In case I<de jure> canonical names differ from that of the Encode
+module, they are always aliased if it ever be implemented. So you can
+safely tell if a given encoding is implemented or not just by passing
+the canonical name.
Because of all the alias issues, and because in the general case
encodings have state, "Encode" uses the encoding object internally
Canonical Aliases Comments & References
----------------------------------------------------------------
+ ascii US-ascii [ECMA]
iso-8859-1 latin1 [ISO]
- US-ascii ascii [ECMA]
- UCS-2 ucs2, iso-10646-1 [IANA, et al]
- UCS-2l
- UTF-8 utf8 [RFC2279]
+ utf8 UTF-8 [RFC2279]
+ UCS-2 ucs2, iso-10646-1, UTF-16LE [IANA, UC]
+ UTF-16LE UCS-2LE [UC]
----------------------------------------------------------------
-=head2 Encode::Byte
+=head2 Encode::Byte -- Extended ASCII
-The following encodings are based single-byte encoding implemented as
-extended ASCII. For most cases it uses \x80-\xff (upper half) to map
-non-ASCII characters.
+Encode::Byte implements most of single-byte encodings except for
+Symbols and EBCDIC. The following encodings are based single-byte
+encoding implemented as extended ASCII. For most cases it uses
+\x80-\xff (upper half) to map non-ASCII characters.
+
+=over 2
+
+=item ISO-8859 and corresponding vendor mappings
+
+Since there are so many, They are presented in table format with
+Languages and corresponding encoding names by vendors. Note the table
+is sorted in order of ISO-8859 and the corresponding vendor mappings
+are slightly different from that of ISO. See
+L<http://czyborra.com/charsets/iso8859.html> for details.
+
+ Lang/Regions ISO/Other Std. DOS Windows Macintosh Others
+ ----------------------------------------------------------------
+ N. America (ASCII) cp437 AdobeStandardEncoding
+ cp863 (DOSCanadaF)
+ W. Europe (iso-8859-1) cp850 cp1252 MacRoman nextstep
+ hp-roman8
+ cp860 (DOSPortuguese)
+ CE. Europe iso-8859-2 cp852 cp1250 MacCentralEurRoman
+ MacCroatian
+ MacRomanian
+ MacRumanian
+ Latin3(*3) iso-8859-3
+ Latin4(*4) iso-8859-4
+ Cyrillics iso-8859-5 cp855 cp1251 MacCyrillic
+ (Also see next section) cp866 MacUkrainian
+ Arabic iso-8859-6 cp864 cp1256 MacArabic
+ cp1006 MacFarsi
+ Greek iso-8859-7 cp737 cp1253 MacGreek
+ cp869 (DOSGreek2)
+ Hebrew iso-8859-8 cp862 cp1255 MacHebrew
+ Turkish iso-8859-9 cp857 cp1254 MacTurkish
+ Nordics iso-8859-10 cp865
+ cp861 MacIcelandic
+ MacSami
+ Thai iso-8859-11 cp874 MacThai
+ (iso-8859-12 is nonexistent. Reserved for Indics?)
+ Baltics iso-8859-13 cp775 cp1257
+ Celtics iso-8859-14
+ Latin9(*15) iso-8859-15
+ Latin10 iso-8859-16
+ Vietnamese viscii cp1258 MacVietnamese
+ ----------------------------------------------------------------
+
+ (*3) Esperanto, Maltese, and Turkish. Turkish is now on 8859-5
+ (*4) Baltics. Now on 8859-10
+ (*9) Nicknamed Latin0; Euro sign as well as French and Finnish
+ letters that are missing from 8859-1 are added.
+
+All cp* are also available as ibm-*, ms-*, and windows-* . See also
+L<http://czyborra.com/charsets/codepages.html>.
+
+Macintosh encodings don't seem to be registered in such entities as
+IANA. "Canonical" names in Encode are based upon Apple's Tech Note
+1150. See L<http://developer.apple.com/technotes/tn/tn1150.html>
+for details
+
+=item KOI8 - De Facto Standard for Cyrillic world
+
+Though ISO-8859 does have ISO-8859, KOI8 series is far more popular
+in the Net. L<Encode> comes with the following KOI charsets. for
+gory details, See <http://czyborra.com/charsets/cyrillic.html> for
+details.
----------------------------------------------------------------
- # ISO 8859 series
- (iso-8859-1 is in built-in)
- iso-8859-2 latin2 [ISO]
- iso-8859-3 latin3 [ISO]
- iso-8859-4 latin4 [ISO]
- iso-8859-5 [ISO]
- iso-8859-6 [ISO]
- iso-8859-7 [ISO]
- iso-8859-8 [ISO]
- iso-8859-9 latin5 [ISO]
- iso-8859-10 latin6 [ISO]
- iso-8859-11
- (iso-8859-12 is nonexistent)
- iso-8859-13 latin7 [ISO]
- iso-8859-14 latin8 [ISO]
- iso-8859-15 latin9 [ISO]
- iso-8859-16 latin10 [ISO]
-
- # Cyrillic
koi8-f
- koi8-r [RFC1489]
+ koi8-r cp878 [RFC1489]
koi8-u [RFC2319]
-
- # Vietnamese
- viscii
- # all cp* are also available as ibm-*, ms-*, and windows-*
- # also see L<http://msdn.microsoft.com/workshop/author/dhtml/reference/charsets/charset4.asp>
- cp1250 WinLatin2
- cp1251 WinCyrillic
- cp1252 WinLatin1
- cp1253 WinGreek
- cp1254 WinTurkiskh
- cp1255 WinHebrew
- cp1256 WinArabic
- cp1257 WinBaltic
- cp1258 WinVietnamese
-
- # Macintosh
- # Also see L<http://developer.apple.com/technotes/tn/tn1150.html>
- MacCentralEurRoman
- MacCroatian
- MacRoman
- MacCyrillic
- MacRomanian
- MacSami
- MacGreek
- MacThai
- MacIceland
- MacTurkish
- MacUkrainian
-
- # More vendor encodings
- nextstep
- gsm0338 # used in GSM handsets
- hp-roman8
- ----------------------------------------------------------------
+=item gsm0338 - Hentai Latin 1
+
+GSM0338 is for GSM handsets. Though it shares alpanumerals with ASCII,
+control character ranges and other parts are mapped very differently,
+presumablly to store Cyrillics. This one is also covered in
+Encode::Byte even thought this one does not comply extended ASCII.
+
+=back
=head2 The CJK: Chinese, Japanese, Korean (Multibyte)
=item Encode::CN -- Continental China
+ Standard DOS/Win Macintosh Comment
----------------------------------------------------------------
- cp936 gbk
- euc-cn gb2312
- gb12345-raw
- gb2312-raw
+ euc-cn MacChineseSimp GB2312 is aliased to this
+ (gbk) cp936 GBK is aliased to to this
+ gb12345-raw GB12345 as is
+ gb2312-raw GB2312 as is
hz
iso-ir-165
----------------------------------------------------------------
=item Encode::JP -- Japan
+ Standard DOS/Win Macintosh Comment/Reference
----------------------------------------------------------------
+ euc-jp
+ shiftjis cp932 macJapanese
7bit-jis jis
- cp932 ms_Kanji
euc-jp ujis
- iso-2022-jp [RFC1468]
- iso-2022-jp-1 [RFC2237]
- macJapan
- shiftjis Shift_JIS, sjis
+ iso-2022-jp [RFC1468]
+ iso-2022-jp-1 [RFC2237]
----------------------------------------------------------------
=item Encode::KR -- Korea
----------------------------------------------------------------
- euc-kr
- cp949 ks_c_5601-1987 x-windows-949 uhc
- iso-2022-kr [RFC1557]
- johab
- ksc5601-raw
+ euc-kr MacKorean [RFC1557]
+ cp949 ks_c_5601-1987 is an alias
+ thereof.
+ iso-2022-kr [RFC1557]
+ johab [KS X 1001:1998, Annex 3]
+ ksc5601-raw KSC5601 as is
----------------------------------------------------------------
=item Encode::TW -- Taiwan
----------------------------------------------------------------
- big5
+ big5 cp950 MacChineseTrad
big5-hkscs
- cp950
----------------------------------------------------------------
=item Encode::HanExtra -- More Chinese via CPAN
=item Encode::EBCDIC
-See perlebcdic for details.
+See L<perlebcdic> for details.
----------------------------------------------------------------
- cp1047
cp37
+ cp500
+ cp875
+ cp1026
+ cp1047
posix-bc
----------------------------------------------------------------
----------------------------------------------------------------
symbol
dingbats
- macDingbats
+ MacDingbats
+ AdobeZdingbat
+ AdobeSymbol
----------------------------------------------------------------
=back
=item ISO-8859-8-1 [Hebrew]
-None of the Encode team knows Hebrew enough. Contribution welcome.
+None of the Encode team knows Hebrew enough (ISO-8859-8, cp1255 and
+MacHebrew are supported because and just because there were mappings
+available at L<http://www.unicode.org/>). Contribution welcome.
=item Thai encoding TCVN
=item Vietnamese encodings VPS
-Ditto.
+Though Jungshik has reported that mozilla supports this encoding, It was too late for us to add one. In future via a separate module. See
+L<http://lxr.mozilla.org/seamonkey/source/intl/uconv/ucvlatin/vps.uf> and
+L<http://lxr.mozilla.org/seamonkey/source/intl/uconv/ucvlatin/vps.ut>
+if you are interested in helping us.
=item various Mac encodings
-The following are unsoported due to the lack of mapping data. "Mac"
-that prepends the encoding names are omitted.
+The following are unsoported due to the lack of mapping data.
+
+ MacArmenian, MacBengali, MacBurmese, MacEthiopic
+ MacExtArabic, MacGeorgian, MacKannada, MacKhmer
+ MacLaotian, MacMalayalam, MacMongolian, MacOriya
+ MacSinhalese, MacTamil, MacTelugu, MacTibetan
+ MacVietnamese
+
+The rest of which already available are based upon the vendor mappings at
+L<http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/> .
+
+=item (Mac) Indic encodings
+
+The maps for the following is available at L<http://www.unicode.org/>
+but remains unsupport because those encordigs need algorithmical
+approach, unsupported by F<enc2xs>
- Arabic, Armenian, Bengali, Burmese
- ChineseSimp, ChineseTrad, Devanagari, Ethiopic, ExtArabic
- Farsi, Georgian, Gujarati, Gurmukhi, Hebrew
- Kannada, Khmer, Korean, Laotian, Malayalam, Mongolian
- Oriya Sinhalese Symbol Tamil Telugu Tibetan Vietnamese
+ MacDevanagari
+ MacGurmukhi
+ MacGujarati
-The rest of which already available are based upon the vendor mapping
-available at L<http://www.unicode.org/>
+For details, please see C<Unicode mapping issues and notes:> at
+L<http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/DEVANAGA.TXT> .
+
+I believe this issue is prevalent not only for Mac Indics but also in
+other Indic encodings but those mentions were the only Indic encodings
+maps that I could find at L<http://www.unicode.org/> .
=back
-=head1 Encoding vs. Charset
+=head1 Encoding vs. Charset -- terminology
-Character encoding (or just "encoding") and Character Set (or just
-"charset") are often used interchangeably but they are different
-concepts.
+We are used to using the term (character) I<encoding> and I<character set>
+interchangeably. But just as using the term byte and character is
+dangerous and should be differenciated when needed, we need to
+differenciate I<encoding> and I<character set>.
-=over 2
+To understand that, it's follow how we make computers grok our character.
+
+=over 4
+
+=item *
-=item Character I<Set> (I<charset> for short)
+First we start with which characters to include. We call this
+collection of characters I<character repertoire>.
-Is a collection of characters in which each character is distinguished
-with unique ID (in most cases, ID is number).
+=item *
-=item Character I<Encoding>
+Then we have to give each character a unique ID so your computer can
+tell the differnce from 'a' to 'A'. This itemized character
+repartoire is now a I<character set>.
-Is a way to represent character set(s) in a stream of bits.
+=item *
+
+If your computer can grow the character set without further
+proccessing, you can go ahead use it. This is called a I<coded
+character set> (CCS) or I<raw character encoding>. ASCII is used this
+way for most cases.
+
+=item *
+
+But in many cases especially multi-byte CJK encodings, you have to
+tweak a little more. Your network connection may not accept any data
+with the Most Significant Bit set, Your computer may not be able to
+tell if a given byte is a whole character or just half of it. So you
+have to I<encode> the character set to use it.
+
+A I<character encoding scheme> (CES) determines how to encode a given
+character set, or a set of multiple character sets. 7bit ISO-2022 is
+an example of CES. You switch between character sets via I<escape
+sequence>.
=back
-A character encoding may contain a single character set
-(i.e. US-ascii) or multiple character sets (i.e. EUC-JP;
-US-ascii, JIS X 0201 Kana, JIS X 0208 and JIS X 0212).
+Technically, or Mathematically speaking, a character set encoded in
+such a CES that maps character by character may form a CCS. EUC is such
+an example. CES of EUC is as follows;
-A character encoding may also encode character set as-is (also called
-a I<raw> encoding. i.e. US-ascii) or processed (i.e. EUC-JP, US-ascii is
-as-is, JIS X 0201 is prepended with \x8E, JIS X 0208 is added by
-0x8080, and JIS X 0212 is added by 0x8080 then prepended with \x8F).
+=over 4
-As the name suggests, the Encode module supports encodings, not
-individual charsets.
+=item *
-However, the word I<charset> is casually used even in Internet
-Assigned Number Authority to actually mean I<encoding>. Encode tries
-to soothe this misconception via aliases. For instance,
-C<gb2312> is aliased to C<euc-cn>, while "raw" encoded version is
-available as C<gb2312-raw>.
+Map ASCII unchanged.
+
+=item *
+
+Map such a character set that consists of 94 or 96 powered by N
+members by adding 0x80 to each byte.
+
+=item *
+
+You can also use 0x8e and 0x8f to tell the following sequence of
+characters belong to yet another character set. each following byte
+is added by 0x80
+
+=back
+
+By carefully looking at at the encoded byte sequence, you may find the
+byte sequence conforms a unique number. In that sense EUC is a CCS
+generated by a CES above from up to four CCS (complicated?). UTF-8
+falls into this category. See L<perlunicode/"UTF-8"> to find how
+UTF-8 maps Unicode to a byte sequence.
+
+You may also find by now why 7bit ISO-2022 cannot conform a CCS. If
+you look at a byte sequence \x21\x21, you can't tell if it is two !'s
+or IDEOGRAPHIC SPACE. EUC maps the latter to \xA1\xA1 so you have no
+trouble between "!!". and " "
=head1 Encoding Classification (by Anton Tagunov and Dan Kogai)
=item *
-To (en|de) code Encodings marked as C<*>, You need C<Encode::HanExtra>
-,available from CPAN.
+To (en|de) code Encodings marked as C<(*)>, You need
+C<Encode::HanExtra>, available from CPAN.
=back
US-ASCII UTF-8 ISO-8859-* KOI8-R
Shift_JIS EUC-JP ISO-2022-JP ISO-2022-JP-1
- EUC-KR Big5
+ EUC-KR Big5 GB2312
+
+are registered to IANA as preferred MIME names and may probably
+be used over the Internet.
-are registered to IANA as preferred MIME names and may probably be used over the Internet.
+C<Shift_JIS> has been officialized by JIS X 0208-1997.
+L<Microsoft-related naming mess> gives details.
-C<Shift_JIS> is no longer Microsft proprietary since it has been
-officialized by JIS X 0208-1997.
+C<GB2312> is the IANA name for C<EUC-CN>.
+See L<Microsoft-related naming mess> for details.
+
+C<GB_2312-80> I<raw> encoding is available as C<gb2312-raw>
+with Encode. See L<Encode::CN -- Continental China> for details.
EUC-CN
+ KOI8-U (http://www.faqs.org/rfcs/rfc2319.html)
-has not been registered with IANA (as of march 2002) but
-seems to be supported by major web browsers. In Encode, GB2312
-is aliased to EUC-CN, with "uncooked" version of GB2312 canonicalized
-as gb2312-raw. See L<Encode::CN> for details.
+have not been registered with IANA (as of March 2002) but
+seem to be supported by major web browsers.
+IANA name for C<EUC-CN> is C<GB2312>.
KS_C_5601-1987
-has been registered to IANA but when they are used, they are
-EUC-coded. Internet community in Korea is not happy with this.
-so C<KS_C_5601-1987> is aliased to C<cp949>, an enhanced version
-of C<euc-kr>, with ksc5601-raw for "uncooked".
+is heavily misused.
+See L<Microsoft-related naming mess> for details.
+
+C<KS_C_5601-1987> I<raw> encoding is available as C<kcs5601-raw>
+with Encode. See L<Encode::KR -- Korea> for details.
UTF-16
- KOI8-U (http://www.faqs.org/rfcs/rfc2319.html)
-are IANA-registered (C<UTF-16> even as a preferred MIME name)
+=for comment
+waiting for comments from Jungshik Shin to soften this - Anton
+
+is a IANA-registered preferred MIME name
but probably should be avoided as encoding for web pages due to
-the lack of browser supports.
+the lack of browser support.
ISO-IR-165 (http://www.faqs.org/rfcs/rfc1345.html)
GBK
is a bit proprietary name.
-=head1 Bookmarks
+=head2 Microsoft-related naming mess
+
+Microsoft products misuse the following names:
=over 2
-=item Assigned Charset Names by IANA
+=item KS_C_5601-1987
-L<http://www.iana.org/assignments/character-sets>
+Microsoft extension to C<EUC-KR>.
-Most of the C<canonical names> in Encode derive from this list
-so you can directly apply the string you have extracted from MIME
-header of mails and we pages.
+Proper name: C<CP949>.
-=item CJK.inf
+See
+http://lists.w3.org/Archives/Public/ietf-charsets/2001AprJun/0033.html
+for details.
-L<http://www.oreilly.com/people/authors/lunde/cjk_inf.html>
+Encode aliases C<KS_C_5601-1987> to C<cp949> to reflect
+this common misusage.
+I<Raw> C<KS_C_5601-1987> encoding is available as C<kcs5601-raw>.
-Somewhat obsolete (last update in 1996), but still useful. Also try
+See L<Encode::KR -- Korea> for details.
-L<ftp://ftp.oreilly.com/pub/examples/nutshell/cjkv/pdf/GB18030_Summary.pdf>
+=item GB2312
-You will find brief info on C<EUC-CN>, C<GBK> and mostly on C<GB 18030>
+Microsoft extension to C<EUC-CN>.
-=item EMCA-035 (eq C<ISO-2022>)
+Proper names: C<CP936>, C<GBK>.
-L<http://www.ecma.ch/ecma1/STAND/ECMA-035.HTM>
+C<GB2312> has been registered in the C<EUC-CN> meaning at
+IANA. This has partially repaired the situation: Microsoft's
+C<GB2312> has become a superset of the official C<GB2312>.
-The very dspecification of ISO-2022 is available from the link above.
+Encode aliases C<GB2312> to C<euc-cn> in full agreement with
+IANA registration. C<cp936> is supported separately.
+I<Raw> C<GB_2312-80> encoding is available as C<kcs5601-raw>.
+
+See L<Encode::CN -- Continental China> for details.
+
+=item Big5
+
+Microsoft extension to C<Big5>.
+
+Proper name: C<CP950>.
+
+Encode separately supports C<Big5> and C<cp950>.
+
+=item Shift_JIS
+
+Microsoft's understanding of C<Shift_JIS>.
+
+JIS has not endorsed the full Microsoft standard however.
+The official C<Shift_JIS> includes only JIS X 0201 and JIS X 0208
+subsets, while Microsoft has always been meaning C<Shift_JIS> to
+encode a wider character repertoire.
+
+As a historical predecessor Microsoft's variant
+probably has more rights for the name, albeit it may be objected
+that Microsoft shouldn't have used JIS as part of the name
+in the first place.
+
+Unabiguous name: C<CP932>.
+
+Encode separately supports C<Shift_JIS> and C<cp932>.
+
+=back
+
+=head1 Glossary
+
+=over 2
+
+=item character repertoire
+
+A collection of unique characters. A I<character> set in the most
+strict sense. At this stage characters are not numberd.
+
+=item coded character set (CCS)
+
+A character set that is mapped in a way computers can use directly.
+Many character encodings including EUC falls in this category.
+
+=item character encoding scheme (CES)
+
+An algorithm to map a character set to a byte sequence. You don't
+have to be able to tell which character set a given byte sequence
+belongs. 7-bit ISO-2022 is a CES but it cannot be a CCS. EUC is an
+example of being both a CCS and CES.
+
+=item EUC
+
+Extended Unix Character. See ISO-2022
+
+=item ISO-2022
+
+A CES that was carefully designed to coexist with ASCII. There are 7
+bit version and 8 bit version. 8 bit version can conform a CCS. EUC
+and ISO-8859 are two examples thereof.
+
+=item UCS
+
+Short for I<Universal Character Set>. When you say just UCS, it means
+I<Unicode>
+
+=item UCS-2
+
+ISO/IEC 10646 encoding form: Universal Character Set coded in two
+octets.
+
+=item Unicode
+
+A Character Set that aims to include all character character
+repertoire of the world. Many character sets in various national as
+well as industorial standards are therefore a subset thereof.
+
+=item UTF
+
+Short for I<Unicode Transformation Format>. Determinse how to map a
+unicode character into byte sequnece.
+
+=item UTF-16
+
+A UTF in 16-bit encoding. Can either be in big endian or little
+endian. Big endian version is called UTF-16BE and little endian
+version is UTF-16LE.
=back
L<Encode::CN>, L<Encode::JP>, L<Encode::KR>, L<Encode::TW>,
L<Encode::EBCDIC>, L<Encode::Symbol>
+=head1 References
+
+=over 2
+
+=item ECMA
+
+European Computer Manufacturers Association
+L<http://www.ecma.ch>
+
+=over 2
+
+=item EMCA-035 (eq C<ISO-2022>)
+
+L<http://www.ecma.ch/ecma1/STAND/ECMA-035.HTM>
+
+The very dspecification of ISO-2022 is available from the link above.
+
+=back
+
+=item IANA
+
+Internet Assigned Numbers Authority
+L<http://www.iana.org/>
+
+=over 2
+
+=item Assigned Charset Names by IANA
+
+L<http://www.iana.org/assignments/character-sets>
+
+Most of the C<canonical names> in Encode derive from this list
+so you can directly apply the string you have extracted from MIME
+header of mails and we pages.
+
+=back
+
+=item ISO
+
+International Organization for Standardization
+L<http://www.iso.ch/>
+
+=item RFC
+
+Request For Comment -- need I say more?
+L<http://www.rfc.net/>
+
+=item UC
+
+Unicode Consortium
+L<http://www.unicode.org/>
+
+=over 2
+
+=item Unicode Glossary
+
+L<http://www.unicode.org/glossary/>
+
+The glossary of this document is based opon this site.
+
+=back
+
+=back
+
+=head2 Other Notable Sites
+
+=over 2
+
+=item czyborra.com
+
+<http://czyborra.com/>
+
+Contains a a lot of useful information, especially gory details of ISO
+vs. vendor mappings.
+
+=item CJK.inf
+
+L<http://www.oreilly.com/people/authors/lunde/cjk_inf.html>
+
+Somewhat obsolete (last update in 1996), but still useful. Also try
+
+L<ftp://ftp.oreilly.com/pub/examples/nutshell/cjkv/pdf/GB18030_Summary.pdf>
+
+You will find brief info on C<EUC-CN>, C<GBK> and mostly on C<GB 18030>
+
+=back
+
=cut
I could not find this page because the hostname doesn't resolve!
projects / p5sagit/p5-mst-13.2.git / blobdiff