X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlunicode.pod;h=d629cabe9f1c120a3db78d1ab4bdff3cb111c4cf;hb=7237d65751f248e676243bc1e148084f323f4838;hp=b8bbc5707cb1adadaa3d721f877c2c073efe5512;hpb=49cb94c67d828cadfe8cac24ae5955cf752eb2df;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index b8bbc57..d629cab 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -4,29 +4,40 @@ perlunicode - Unicode support in Perl =head1 DESCRIPTION -=head2 Important Caveat +=head2 Important Caveats -WARNING: The implementation of Unicode support in Perl is incomplete. +WARNING: While the implementation of Unicode support in Perl is now fairly +complete it is still evolving to some extent. -The following areas need further work. +In particular the way Unicode is handled on EBCDIC platforms is still rather +experimental. On such a platform references to UTF-8 encoding in this +document and elsewhere should be read as meaning UTF-EBCDIC as specified +in Unicode Technical Report 16 unless ASCII vs EBCDIC issues are specifically +discussed. There is no C pragma or ":utfebcdic" layer, rather +"utf8" and ":utf8" are re-used to mean platform's "natural" 8-bit encoding +of Unicode. See L for more discussion of the issues. + +The following areas are still under development. =over 4 =item Input and Output Disciplines -There is currently no easy way to mark data read from a file or other -external source as being utf8. This will be one of the major areas of -focus in the near future. Unfortunately it is unlikely that the Perl -5.6 and earlier will ever gain this capability. +A filehandle can be marked as containing perl's internal Unicode encoding +(UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer. +Other encodings can be converted to perl's encoding on input, or from +perl's encoding on output by use of the ":encoding()" layer. +There is not yet a clean way to mark the perl source itself as being +in an particular encoding. =item Regular Expressions -The existing regular expression compiler does not produce polymorphic -opcodes. This means that the determination on whether to match Unicode -characters is made when the pattern is compiled, based on whether the -pattern contains Unicode characters, and not when the matching happens -at run time. This needs to be changed to adaptively match Unicode if -the string to be matched is Unicode. +The regular expression compiler does now attempt to produce +polymorphic opcodes. That is the pattern should now adapt to the data +and automatically switch to the Unicode character scheme when presented +with Unicode data, or a traditional byte scheme when presented with +byte data. The implementation is still new and (particularly on +EBCDIC platforms) may need further work. =item C still needed to enable a few features @@ -36,7 +47,8 @@ normally be used. However, as a compatibility measure, this pragma must be explicitly used to enable recognition of UTF-8 encoded literals and identifiers in the -source text. +source text on ASCII based machines or recognize UTF-EBCDIC encoded literals +and identifiers on EBCDIC based machines. =back @@ -44,7 +56,7 @@ source text. Beginning with version 5.6, Perl uses logically wide characters to represent strings internally. This internal representation of strings -uses the UTF-8 encoding. +uses either the UTF-8 or the UTF-EBCDIC encoding. In future, Perl-level operations can be expected to work with characters rather than bytes, in general. @@ -67,14 +79,13 @@ or from literals and constants in the source text. If the C<-C> command line switch is used, (or the ${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls will use the corresponding wide character APIs. This is currently only implemented -on Windows as other platforms do not have a unified way of handling -wide character APIs. +on Windows since UNIXes lack API standard on this area. Regardless of the above, the C pragma can always be used to force byte semantics in a particular lexical scope. See L. The C pragma is primarily a compatibility device that enables -recognition of UTF-8 in literals encountered by the parser. It may also +recognition of UTF-(8|EBCDIC) in literals encountered by the parser. It may also be used for enabling some of the more experimental Unicode support features. Note that this pragma is only required until a future version of Perl in which character semantics will become the default. This pragma may @@ -94,6 +105,8 @@ bytes change to operating on characters. For ASCII data this makes no difference, because UTF-8 stores ASCII in single bytes, but for any character greater than C, the character may be stored in a sequence of two or more bytes, all of which have the high bit set. +For C1 controls or Latin 1 characters on an EBCDIC platform the character +may be stored in a UTF-EBCDIC multi byte sequence. But by and large, the user need not worry about this, because Perl hides it from the user. A character in Perl is logically just a number ranging from 0 to 2**32 or so. Larger characters encode to longer @@ -112,9 +125,9 @@ Strings and patterns may contain characters that have an ordinal value larger than 255. Presuming you use a Unicode editor to edit your program, such characters -will typically occur directly within the literal strings as UTF-8 +will typically occur directly within the literal strings as UTF-(8|EBCDIC) characters, but you can also specify a particular character with an -extension of the C<\x> notation. UTF-8 characters are specified by +extension of the C<\x> notation. UTF-X characters are specified by putting the hexadecimal code within curlies after the C<\x>. For instance, a Unicode smiley face is C<\x{263A}>. @@ -129,7 +142,8 @@ attempt to canonicalize variable names for you.) Regular expressions match characters instead of bytes. For instance, "." matches a character instead of a byte. (However, the C<\C> pattern -is available to force a match a single byte ("C" in C, hence C<\C>).) +is provided to force a match a single byte ("C" in C, hence +C<\C>).) =item * @@ -144,9 +158,179 @@ Named Unicode properties and block ranges make be used as character classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't match property) constructs. For instance, C<\p{Lu}> matches any character with the Unicode uppercase property, while C<\p{M}> matches -any mark character. Single letter properties may omit the brackets, so -that can be written C<\pM> also. Many predefined character classes are -available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>. +any mark character. Single letter properties may omit the brackets, +so that can be written C<\pM> also. Many predefined character classes +are available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>. The +names of the C classes are the official Unicode block names but +with all non-alphanumeric characters removed, for example the block +name C<"Latin-1 Supplement"> becomes C<\p{InLatin1Supplement}>. + +Here is the list as of Unicode 3.1.0 (the two-letter classes) and +Perl 5.8.0 (the one-letter classes): + + L Letter + Lu Letter, Uppercase + Ll Letter, Lowercase + Lt Letter, Titlecase + Lm Letter, Modifier + Lo Letter, Other + M Mark + Mn Mark, Non-Spacing + Mc Mark, Spacing Combining + Me Mark, Enclosing + N Number + Nd Number, Decimal Digit + Nl Number, Letter + No Number, Other + P Punctuation + Pc Punctuation, Connector + Pd Punctuation, Dash + Ps Punctuation, Open + Pe Punctuation, Close + Pi Punctuation, Initial quote + (may behave like Ps or Pe depending on usage) + Pf Punctuation, Final quote + (may behave like Ps or Pe depending on usage) + Po Punctuation, Other + S Symbol + Sm Symbol, Math + Sc Symbol, Currency + Sk Symbol, Modifier + So Symbol, Other + Z Separator + Zs Separator, Space + Zl Separator, Line + Zp Separator, Paragraph + C Other + Cc Other, Control + Cf Other, Format + Cs Other, Surrogate + Co Other, Private Use + Cn Other, Not Assigned (Unicode defines no Cn characters) + +Additionally, because scripts differ in their directionality +(for example Hebrew is written right to left), all characters +have their directionality defined: + + BidiL Left-to-Right + BidiLRE Left-to-Right Embedding + BidiLRO Left-to-Right Override + BidiR Right-to-Left + BidiAL Right-to-Left Arabic + BidiRLE Right-to-Left Embedding + BidiRLO Right-to-Left Override + BidiPDF Pop Directional Format + BidiEN European Number + BidiES European Number Separator + BidiET European Number Terminator + BidiAN Arabic Number + BidiCS Common Number Separator + BidiNSM Non-Spacing Mark + BidiBN Boundary Neutral + BidiB Paragraph Separator + BidiS Segment Separator + BidiWS Whitespace + BidiON Other Neutrals + +The blocks available for C<\p{InBlock}> and C<\P{InBlock}>, for +example \p{InCyrillic>, are as follows: + + BasicLatin + Latin1Supplement + LatinExtendedA + LatinExtendedB + IPAExtensions + SpacingModifierLetters + CombiningDiacriticalMarks + Greek + Cyrillic + Armenian + Hebrew + Arabic + Syriac + Thaana + Devanagari + Bengali + Gurmukhi + Gujarati + Oriya + Tamil + Telugu + Kannada + Malayalam + Sinhala + Thai + Lao + Tibetan + Myanmar + Georgian + HangulJamo + Ethiopic + Cherokee + UnifiedCanadianAboriginalSyllabics + Ogham + Runic + Khmer + Mongolian + LatinExtendedAdditional + GreekExtended + GeneralPunctuation + SuperscriptsandSubscripts + CurrencySymbols + CombiningMarksforSymbols + LetterlikeSymbols + NumberForms + Arrows + MathematicalOperators + MiscellaneousTechnical + ControlPictures + OpticalCharacterRecognition + EnclosedAlphanumerics + BoxDrawing + BlockElements + GeometricShapes + MiscellaneousSymbols + Dingbats + BraillePatterns + CJKRadicalsSupplement + KangxiRadicals + IdeographicDescriptionCharacters + CJKSymbolsandPunctuation + Hiragana + Katakana + Bopomofo + HangulCompatibilityJamo + Kanbun + BopomofoExtended + EnclosedCJKLettersandMonths + CJKCompatibility + CJKUnifiedIdeographsExtensionA + CJKUnifiedIdeographs + YiSyllables + YiRadicals + HangulSyllables + HighSurrogates + HighPrivateUseSurrogates + LowSurrogates + PrivateUse + CJKCompatibilityIdeographs + AlphabeticPresentationForms + ArabicPresentationFormsA + CombiningHalfMarks + CJKCompatibilityForms + SmallFormVariants + ArabicPresentationFormsB + Specials + HalfwidthandFullwidthForms + OldItalic + Gothic + Deseret + ByzantineMusicalSymbols + MusicalSymbols + MathematicalAlphanumericSymbols + CJKUnifiedIdeographsExtensionB + CJKCompatibilityIdeographsSupplement + Tags =item * @@ -217,16 +401,13 @@ And finally, C reverses by character rather than by byte. =head2 Character encodings for input and output -This feature is in the process of getting implemented. - -(For Perl 5.6 and earlier the support is unlikely to get integrated -to the core language and some external module will be required.) +See L. =head1 CAVEATS As of yet, there is no method for automatically coercing input and -output to some encoding other than UTF-8. This is planned in the near -future, however. +output to some encoding other than UTF-8 or UTF-EBCDIC. This is planned +in the near future, however. Whether an arbitrary piece of data will be treated as "characters" or "bytes" by internal operations cannot be divined at the current time. @@ -239,6 +420,6 @@ tend to run slower. Avoidance of locales is strongly encouraged. =head1 SEE ALSO -L, L, L +L, L, L, L =cut