X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlunicode.pod;h=f429be74b27afbd098b222a29f2db0f3b011fb29;hb=ad0029c435199eaf70c06265f639c1af50f36906;hp=c9954d8e96a81e4a63effb485e8b22f9eeb08230;hpb=383e7cdd17eec132ddb7b17dd6275f3153cbe989;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index c9954d8..f429be7 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -4,38 +4,52 @@ perlunicode - Unicode support in Perl =head1 DESCRIPTION -=head2 Important Caveat +=head2 Important Caveats -WARNING: The implementation of Unicode support in Perl is incomplete. +WARNING: While the implementation of Unicode support in Perl is now fairly +complete it is still evolving to some extent. -The following areas need further work. +In particular the way Unicode is handled on EBCDIC platforms is still +rather experimental. On such a platform references to UTF-8 encoding +in this document and elsewhere should be read as meaning UTF-EBCDIC as +specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues +are specifically discussed. There is no C pragma or +":utfebcdic" layer, rather "utf8" and ":utf8" are re-used to mean +platform's "natural" 8-bit encoding of Unicode. See L for +more discussion of the issues. -=over +The following areas are still under development. + +=over 4 =item Input and Output Disciplines -There is currently no easy way to mark data read from a file or other -external source as being utf8. This will be one of the major areas of -focus in the near future. +A filehandle can be marked as containing perl's internal Unicode +encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer. +Other encodings can be converted to perl's encoding on input, or from +perl's encoding on output by use of the ":encoding()" layer. There is +not yet a clean way to mark the Perl source itself as being in an +particular encoding. =item Regular Expressions -The existing regular expression compiler does not produce polymorphic -opcodes. This means that the determination on whether to match Unicode -characters is made when the pattern is compiled, based on whether the -pattern contains Unicode characters, and not when the matching happens -at run time. This needs to be changed to adaptively match Unicode if -the string to be matched is Unicode. +The regular expression compiler does now attempt to produce +polymorphic opcodes. That is the pattern should now adapt to the data +and automatically switch to the Unicode character scheme when +presented with Unicode data, or a traditional byte scheme when +presented with byte data. The implementation is still new and +(particularly on EBCDIC platforms) may need further work. -=item C still needed to enable a few features +=item C still needed to enable UTF-8/UTF-EBCDIC in scripts -The C pragma implements the tables used for Unicode support. These -tables are automatically loaded on demand, so the C pragma need not -normally be used. +The C pragma implements the tables used for Unicode support. +These tables are automatically loaded on demand, so the C pragma +need not normally be used. -However, as a compatibility measure, this pragma must be explicitly used -to enable recognition of UTF-8 encoded literals and identifiers in the -source text. +However, as a compatibility measure, this pragma must be explicitly +used to enable recognition of UTF-8 in the Perl scripts themselves on +ASCII based machines or recognize UTF-EBCDIC on EBCDIC based machines. +B is needed>. =back @@ -43,18 +57,18 @@ source text. Beginning with version 5.6, Perl uses logically wide characters to represent strings internally. This internal representation of strings -uses the UTF-8 encoding. +uses either the UTF-8 or the UTF-EBCDIC encoding. -In future, Perl-level operations can be expected to work with characters -rather than bytes, in general. +In future, Perl-level operations can be expected to work with +characters rather than bytes, in general. -However, as strictly an interim compatibility measure, Perl v5.6 aims to -provide a safe migration path from byte semantics to character semantics -for programs. For operations where Perl can unambiguously decide that the -input data is characters, Perl now switches to character semantics. -For operations where this determination cannot be made without additional -information from the user, Perl decides in favor of compatibility, and -chooses to use byte semantics. +However, as strictly an interim compatibility measure, Perl aims to +provide a safe migration path from byte semantics to character +semantics for programs. For operations where Perl can unambiguously +decide that the input data is characters, Perl now switches to +character semantics. For operations where this determination cannot +be made without additional information from the user, Perl decides in +favor of compatibility, and chooses to use byte semantics. This behavior preserves compatibility with earlier versions of Perl, which allowed byte semantics in Perl operations, but only as long as @@ -63,20 +77,21 @@ character data. Such data may come from filehandles, from calls to external programs, from information provided by the system (such as %ENV), or from literals and constants in the source text. -If the C<-C> command line switch is used, (or the ${^WIDE_SYSTEM_CALLS} -global flag is set to C<1>), all system calls will use the -corresponding wide character APIs. This is currently only implemented -on Windows. +If the C<-C> command line switch is used, (or the +${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls +will use the corresponding wide character APIs. Note that this is +currently only implemented on Windows since other platforms API +standard on this area. -Regardless of the above, the C pragma can always be used to force -byte semantics in a particular lexical scope. See L. +Regardless of the above, the C pragma can always be used to +force byte semantics in a particular lexical scope. See L. The C pragma is primarily a compatibility device that enables -recognition of UTF-8 in literals encountered by the parser. It may also -be used for enabling some of the more experimental Unicode support features. -Note that this pragma is only required until a future version of Perl -in which character semantics will become the default. This pragma may -then become a no-op. See L. +recognition of UTF-(8|EBCDIC) in literals encountered by the parser. +It may also be used for enabling some of the more experimental Unicode +support features. Note that this pragma is only required until a +future version of Perl in which character semantics will become the +default. This pragma may then become a no-op. See L. Unless mentioned otherwise, Perl operators will use character semantics when they are dealing with Unicode data, and byte semantics otherwise. @@ -88,15 +103,18 @@ apply; otherwise, byte semantics are in effect. To force byte semantics on Unicode data, the C pragma should be used. Under character semantics, many operations that formerly operated on -bytes change to operating on characters. For ASCII data this makes -no difference, because UTF-8 stores ASCII in single bytes, but for -any character greater than C, the character may be stored in +bytes change to operating on characters. For ASCII data this makes no +difference, because UTF-8 stores ASCII in single bytes, but for any +character greater than C, the character B be stored in a sequence of two or more bytes, all of which have the high bit set. -But by and large, the user need not worry about this, because Perl -hides it from the user. A character in Perl is logically just a number -ranging from 0 to 2**32 or so. Larger characters encode to longer -sequences of bytes internally, but again, this is just an internal -detail which is hidden at the Perl level. + +For C1 controls or Latin 1 characters on an EBCDIC platform the +character may be stored in a UTF-EBCDIC multi byte sequence. But by +and large, the user need not worry about this, because Perl hides it +from the user. A character in Perl is logically just a number ranging +from 0 to 2**32 or so. Larger characters encode to longer sequences +of bytes internally, but again, this is just an internal detail which +is hidden at the Perl level. =head2 Effects of character semantics @@ -109,33 +127,33 @@ Character semantics have the following effects: Strings and patterns may contain characters that have an ordinal value larger than 255. -Presuming you use a Unicode editor to edit your program, such characters -will typically occur directly within the literal strings as UTF-8 -characters, but you can also specify a particular character with an -extension of the C<\x> notation. UTF-8 characters are specified by -putting the hexadecimal code within curlies after the C<\x>. For instance, -a Unicode smiley face is C<\x{263A}>. +Presuming you use a Unicode editor to edit your program, such +characters will typically occur directly within the literal strings as +UTF-8 (or UTF-EBCDIC on EBCDIC platforms) characters, but you can also +specify a particular character with an extension of the C<\x> +notation. UTF-X characters are specified by putting the hexadecimal +code within curlies after the C<\x>. For instance, a Unicode smiley +face is C<\x{263A}>. =item * Identifiers within the Perl script may contain Unicode alphanumeric characters, including ideographs. (You are currently on your own when -it comes to using the canonical forms of characters--Perl doesn't (yet) -attempt to canonicalize variable names for you.) +it comes to using the canonical forms of characters--Perl doesn't +(yet) attempt to canonicalize variable names for you.) =item * Regular expressions match characters instead of bytes. For instance, "." matches a character instead of a byte. (However, the C<\C> pattern -is provided to force a match a single byte ("C" in C, hence -C<\C>).) +is provided to force a match a single byte ("C" in C, hence C<\C>).) =item * Character classes in regular expressions match characters instead of bytes, and match against the character properties specified in the -Unicode properties database. So C<\w> can be used to match an ideograph, -for instance. +Unicode properties database. So C<\w> can be used to match an +ideograph, for instance. =item * @@ -143,9 +161,248 @@ Named Unicode properties and block ranges make be used as character classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't match property) constructs. For instance, C<\p{Lu}> matches any character with the Unicode uppercase property, while C<\p{M}> matches -any mark character. Single letter properties may omit the brackets, so -that can be written C<\pM> also. Many predefined character classes are -available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>. +any mark character. Single letter properties may omit the brackets, +so that can be written C<\pM> also. Many predefined character classes +are available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>. The +names of the C classes are the official Unicode script and block +names but with all non-alphanumeric characters removed, for example +the block name C<"Latin-1 Supplement"> becomes C<\p{InLatin1Supplement}>. + +Here is the list as of Unicode 3.1.0 (the two-letter classes) and +as defined by Perl (the one-letter classes) (in Unicode materials +what Perl calls C is often called C): + + L Letter + Lu Letter, Uppercase + Ll Letter, Lowercase + Lt Letter, Titlecase + Lm Letter, Modifier + Lo Letter, Other + M Mark + Mn Mark, Non-Spacing + Mc Mark, Spacing Combining + Me Mark, Enclosing + N Number + Nd Number, Decimal Digit + Nl Number, Letter + No Number, Other + P Punctuation + Pc Punctuation, Connector + Pd Punctuation, Dash + Ps Punctuation, Open + Pe Punctuation, Close + Pi Punctuation, Initial quote + (may behave like Ps or Pe depending on usage) + Pf Punctuation, Final quote + (may behave like Ps or Pe depending on usage) + Po Punctuation, Other + S Symbol + Sm Symbol, Math + Sc Symbol, Currency + Sk Symbol, Modifier + So Symbol, Other + Z Separator + Zs Separator, Space + Zl Separator, Line + Zp Separator, Paragraph + C Other + Cc Other, Control + Cf Other, Format + Cs Other, Surrogate + Co Other, Private Use + Cn Other, Not Assigned (Unicode defines no Cn characters) + +Additionally, because scripts differ in their directionality +(for example Hebrew is written right to left), all characters +have their directionality defined: + + BidiL Left-to-Right + BidiLRE Left-to-Right Embedding + BidiLRO Left-to-Right Override + BidiR Right-to-Left + BidiAL Right-to-Left Arabic + BidiRLE Right-to-Left Embedding + BidiRLO Right-to-Left Override + BidiPDF Pop Directional Format + BidiEN European Number + BidiES European Number Separator + BidiET European Number Terminator + BidiAN Arabic Number + BidiCS Common Number Separator + BidiNSM Non-Spacing Mark + BidiBN Boundary Neutral + BidiB Paragraph Separator + BidiS Segment Separator + BidiWS Whitespace + BidiON Other Neutrals + +=head2 Scripts + +The scripts available for C<\p{In...}> and C<\P{In...}>, for example +\p{InCyrillic>, are as follows, for example C<\p{InLatin}> or C<\P{InHan}>: + + Latin + Greek + Cyrillic + Armenian + Hebrew + Arabic + Syriac + Thaana + Devanagari + Bengali + Gurmukhi + Gujarati + Oriya + Tamil + Telugu + Kannada + Malayalam + Sinhala + Thai + Lao + Tibetan + Myanmar + Georgian + Hangul + Ethiopic + Cherokee + CanadianAboriginal + Ogham + Runic + Khmer + Mongolian + Hiragana + Katakana + Bopomofo + Han + Yi + OldItalic + Gothic + Deseret + Inherited + +=head2 Blocks + +In addition to B, Unicode also defines B of +characters. The difference between scripts and blocks is that the +former concept is closer to natural languages, while the latter +concept is more an artificial grouping based on groups of 256 Unicode +characters. For example, the C script contains letters from +many blocks, but it does not contain all the characters from those +blocks, it does not for example contain digits. + +For more about scripts see the UTR #24: +http://www.unicode.org/unicode/reports/tr24/ +For more about blocks see +http://www.unicode.org/Public/UNIDATA/Blocks.txt + +Because there are overlaps in naming (there are, for example, both +a script called C and a block called C, the block +version has C appended to its name, C<\p{InKatakanaBlock}>. + +Notice that this definition was introduced in Perl 5.8.0: in Perl +5.6.0 only the blocks were used; in Perl 5.8.0 scripts became the +preferential character class definition; this meant that the +definitions of some character classes changed (the ones in the +below list that have the C appended). + + BasicLatin + Latin1Supplement + LatinExtendedA + LatinExtendedB + IPAExtensions + SpacingModifierLetters + CombiningDiacriticalMarks + GreekBlock + CyrillicBlock + ArmenianBlock + HebrewBlock + ArabicBlock + SyriacBlock + ThaanaBlock + DevanagariBlock + BengaliBlock + GurmukhiBlock + GujaratiBlock + OriyaBlock + TamilBlock + TeluguBlock + KannadaBlock + MalayalamBlock + SinhalaBlock + ThaiBlock + LaoBlock + TibetanBlock + MyanmarBlock + GeorgianBlock + HangulJamo + EthiopicBlock + CherokeeBlock + UnifiedCanadianAboriginalSyllabics + OghamBlock + RunicBlock + KhmerBlock + MongolianBlock + LatinExtendedAdditional + GreekExtended + GeneralPunctuation + SuperscriptsandSubscripts + CurrencySymbols + CombiningMarksforSymbols + LetterlikeSymbols + NumberForms + Arrows + MathematicalOperators + MiscellaneousTechnical + ControlPictures + OpticalCharacterRecognition + EnclosedAlphanumerics + BoxDrawing + BlockElements + GeometricShapes + MiscellaneousSymbols + Dingbats + BraillePatterns + CJKRadicalsSupplement + KangxiRadicals + IdeographicDescriptionCharacters + CJKSymbolsandPunctuation + HiraganaBlock + KatakanaBlock + BopomofoBlock + HangulCompatibilityJamo + Kanbun + BopomofoExtended + EnclosedCJKLettersandMonths + CJKCompatibility + CJKUnifiedIdeographsExtensionA + CJKUnifiedIdeographs + YiSyllables + YiRadicals + HangulSyllables + HighSurrogates + HighPrivateUseSurrogates + LowSurrogates + PrivateUse + CJKCompatibilityIdeographs + AlphabeticPresentationForms + ArabicPresentationFormsA + CombiningHalfMarks + CJKCompatibilityForms + SmallFormVariants + ArabicPresentationFormsB + Specials + HalfwidthandFullwidthForms + OldItalicBlock + GothicBlock + DeseretBlock + ByzantineMusicalSymbols + MusicalSymbols + MathematicalAlphanumericSymbols + CJKUnifiedIdeographsExtensionB + CJKCompatibilityIdeographsSupplement + Tags =item * @@ -173,13 +430,13 @@ sequences have the same semantics. =item * Most operators that deal with positions or lengths in the string will -automatically switch to using character positions, including C, -C, C, C, C, C, -C, and C. Operators that specifically don't switch -include C, C, and C. Operators that really -don't care include C, as well as any other operator that -treats a string as a bucket of bits, such as C, and the -operators dealing with filenames. +automatically switch to using character positions, including +C, C, C, C, C, +C, C, and C. Operators that +specifically don't switch include C, C, and +C. Operators that really don't care include C, as +well as any other operator that treats a string as a bucket of bits, +such as C, and the operators dealing with filenames. =item * @@ -198,19 +455,31 @@ byte-oriented C and C under utf8. =item * +The bit string operators C<& | ^ ~> can operate on character data. +However, for backward compatibility reasons (bit string operations +when the characters all are less than 256 in ordinal value) one should +not mix C<~> (the bit complement) and characters both less than 256 and +equal or greater than 256. Most importantly, the DeMorgan's laws +(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold. +Another way to look at this is that the complement cannot return +B the 8-bit (byte) wide bit complement B the full character +wide bit complement. + +=item * + And finally, C reverses by character rather than by byte. =back =head2 Character encodings for input and output -[XXX: This feature is not yet implemented.] +See L. =head1 CAVEATS As of yet, there is no method for automatically coercing input and -output to some encoding other than UTF-8. This is planned in the near -future, however. +output to some encoding other than UTF-8 or UTF-EBCDIC. This is planned +in the near future, however. Whether an arbitrary piece of data will be treated as "characters" or "bytes" by internal operations cannot be divined at the current time. @@ -223,6 +492,6 @@ tend to run slower. Avoidance of locales is strongly encouraged. =head1 SEE ALSO -L, L, L +L, L, L, L =cut