=head1 DESCRIPTION
-WARNING: The implementation of Unicode support in Perl is incomplete.
-Expect sudden and unannounced changes!
+=head2 Important Caveats
-Beginning with version 5.6, Perl uses logically wide characters to
-represent strings internally. This internal representation of strings
-uses the UTF-8 encoding.
+WARNING: While the implementation of Unicode support in Perl is now fairly
+complete it is still evolving to some extent.
+
+In particular the way Unicode is handled on EBCDIC platforms is still
+rather experimental. On such a platform references to UTF-8 encoding
+in this document and elsewhere should be read as meaning UTF-EBCDIC as
+specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
+are specifically discussed. There is no C<utfebcdic> pragma or
+":utfebcdic" layer, rather "utf8" and ":utf8" are re-used to mean
+platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic> for
+more discussion of the issues.
+
+The following areas are still under development.
+
+=over 4
+
+=item Input and Output Disciplines
+
+A filehandle can be marked as containing perl's internal Unicode
+encoding (UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
+Other encodings can be converted to perl's encoding on input, or from
+perl's encoding on output by use of the ":encoding()" layer. There is
+not yet a clean way to mark the Perl source itself as being in an
+particular encoding.
+
+=item Regular Expressions
-In future, Perl-level operations will expect to work with characters
-rather than bytes, in general.
+The regular expression compiler does now attempt to produce
+polymorphic opcodes. That is the pattern should now adapt to the data
+and automatically switch to the Unicode character scheme when
+presented with Unicode data, or a traditional byte scheme when
+presented with byte data. The implementation is still new and
+(particularly on EBCDIC platforms) may need further work.
-However, Perl v5.6 aims to provide a safe migration path from byte
-semantics to character semantics for programs. To preserve compatibility
-with earlier versions of Perl which allowed byte semantics in Perl
-operations (owing to the fact that the internal representation for
-characters was in bytes) byte semantics will continue to be in effect
-until a the C<utf8> pragma is used in the C<main> package, or the C<$^U>
-global flag is explicitly set.
+=item C<use utf8> still needed to enable a few features
+
+The C<utf8> pragma implements the tables used for Unicode support.
+These tables are automatically loaded on demand, so the C<utf8> pragma
+need not normally be used.
+
+However, as a compatibility measure, this pragma must be explicitly
+used to enable recognition of UTF-8 encoded literals and identifiers
+in the source text on ASCII based machines or recognize UTF-EBCDIC
+encoded literals and identifiers on EBCDIC based machines.
+
+=back
+
+=head2 Byte and Character semantics
+
+Beginning with version 5.6, Perl uses logically wide characters to
+represent strings internally. This internal representation of strings
+uses either the UTF-8 or the UTF-EBCDIC encoding.
+
+In future, Perl-level operations can be expected to work with
+characters rather than bytes, in general.
+
+However, as strictly an interim compatibility measure, Perl aims to
+provide a safe migration path from byte semantics to character
+semantics for programs. For operations where Perl can unambiguously
+decide that the input data is characters, Perl now switches to
+character semantics. For operations where this determination cannot
+be made without additional information from the user, Perl decides in
+favor of compatibility, and chooses to use byte semantics.
+
+This behavior preserves compatibility with earlier versions of Perl,
+which allowed byte semantics in Perl operations, but only as long as
+none of the program's inputs are marked as being as source of Unicode
+character data. Such data may come from filehandles, from calls to
+external programs, from information provided by the system (such as %ENV),
+or from literals and constants in the source text.
+
+If the C<-C> command line switch is used, (or the
+${^WIDE_SYSTEM_CALLS} global flag is set to C<1>), all system calls
+will use the corresponding wide character APIs. Note that this is
+currently only implemented on Windows since other platforms API
+standard on this area.
+
+Regardless of the above, the C<bytes> pragma can always be used to
+force byte semantics in a particular lexical scope. See L<bytes>.
+
+The C<utf8> pragma is primarily a compatibility device that enables
+recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
+It may also be used for enabling some of the more experimental Unicode
+support features. Note that this pragma is only required until a
+future version of Perl in which character semantics will become the
+default. This pragma may then become a no-op. See L<utf8>.
+
+Unless mentioned otherwise, Perl operators will use character semantics
+when they are dealing with Unicode data, and byte semantics otherwise.
+Thus, character semantics for these operations apply transparently; if
+the input data came from a Unicode source (for example, by adding a
+character encoding discipline to the filehandle whence it came, or a
+literal UTF-8 string constant in the program), character semantics
+apply; otherwise, byte semantics are in effect. To force byte semantics
+on Unicode data, the C<bytes> pragma should be used.
Under character semantics, many operations that formerly operated on
-bytes change to operating on characters. For ASCII data this makes
-no difference, because UTF-8 stores ASCII in single bytes, but for
-any character greater than C<chr(127)>, the character is stored in
+bytes change to operating on characters. For ASCII data this makes no
+difference, because UTF-8 stores ASCII in single bytes, but for any
+character greater than C<chr(127)>, the character B<may> be stored in
a sequence of two or more bytes, all of which have the high bit set.
-But by and large, the user need not worry about this, because Perl
-hides it from the user. A character in Perl is logically just a number
-ranging from 0 to 2**32 or so. Larger characters encode to longer
-sequences of bytes internally, but again, this is just an internal
-detail which is hidden at the Perl level.
-
-The C<byte> pragma can be used to force byte semantics in a particular
-lexical scope. See L<byte>.
-
-The C<utf8> pragma is a compatibility device to enables recognition
-of UTF-8 in literals encountered by the parser. It is also used
-for enabling some experimental Unicode support features. Note that
-this pragma is only required until a future version of Perl in which
-character semantics will become the default. This pragma may then
-become a no-op. See L<utf8>.
+
+For C1 controls or Latin 1 characters on an EBCDIC platform the
+character may be stored in a UTF-EBCDIC multi byte sequence. But by
+and large, the user need not worry about this, because Perl hides it
+from the user. A character in Perl is logically just a number ranging
+from 0 to 2**32 or so. Larger characters encode to longer sequences
+of bytes internally, but again, this is just an internal detail which
+is hidden at the Perl level.
+
+=head2 Effects of character semantics
Character semantics have the following effects:
=item *
Strings and patterns may contain characters that have an ordinal value
-larger than 255. In Perl v5.6, this is only enabled if the lexical
-scope has a C<use utf8> declaration (due to compatibility needs) but
-future versions may enable this by default.
-
-Presuming you use a Unicode editor to edit your program, such characters
-will typically occur directly within the literal strings as UTF-8
-characters, but you can also specify a particular character with an
-extension of the C<\x> notation. UTF-8 characters are specified by
-putting the hexadecimal code within curlies after the C<\x>. For instance,
-a Unicode smiley face is C<\x{263A}>. A character in the Latin-1 range
-(128..255) should be written C<\x{ab}> rather than C<\xab>, since the
-former will turn into a two-byte UTF-8 code, while the latter will
-continue to be interpreted as generating a 8-bit byte rather than a
-character. In fact, if C<-w> is turned on, it will produce a warning
-that you might be generating invalid UTF-8.
+larger than 255.
+
+Presuming you use a Unicode editor to edit your program, such
+characters will typically occur directly within the literal strings as
+UTF-8 (or UTF-EBCDIC on EBCDIC platforms) characters, but you can also
+specify a particular character with an extension of the C<\x>
+notation. UTF-X characters are specified by putting the hexadecimal
+code within curlies after the C<\x>. For instance, a Unicode smiley
+face is C<\x{263A}>.
=item *
Identifiers within the Perl script may contain Unicode alphanumeric
characters, including ideographs. (You are currently on your own when
-it comes to using the canonical forms of characters--Perl doesn't (yet)
-attempt to canonicalize variable names for you.)
-
-This also needs C<use utf8> currently. [XXX: Why? High-bit chars were
-syntax errors when they occurred within identifiers in previous versions,
-so this should be enabled by default.]
+it comes to using the canonical forms of characters--Perl doesn't
+(yet) attempt to canonicalize variable names for you.)
=item *
Regular expressions match characters instead of bytes. For instance,
"." matches a character instead of a byte. (However, the C<\C> pattern
-is provided to force a match a single byte ("C<char>" in C, hence
-C<\C>).)
-
-Unicode support in regular expressions needs C<use utf8> currently.
-[XXX: Because the SWASH routines need to be loaded. And the RE engine
-appears to need an overhaul to Unicode by default anyway.]
+is provided to force a match a single byte ("C<char>" in C, hence C<\C>).)
=item *
Character classes in regular expressions match characters instead of
bytes, and match against the character properties specified in the
-Unicode properties database. So C<\w> can be used to match an ideograph,
-for instance.
-
-C<use utf8> is needed to enable this. See above.
+Unicode properties database. So C<\w> can be used to match an
+ideograph, for instance.
=item *
classes via the new C<\p{}> (matches property) and C<\P{}> (doesn't
match property) constructs. For instance, C<\p{Lu}> matches any
character with the Unicode uppercase property, while C<\p{M}> matches
-any mark character. Single letter properties may omit the brackets, so
-that can be written C<\pM> also. Many predefined character classes are
-available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>.
-
-C<use utf8> is needed to enable this. See above.
+any mark character. Single letter properties may omit the brackets,
+so that can be written C<\pM> also. Many predefined character classes
+are available, such as C<\p{IsMirrored}> and C<\p{InTibetan}>. The
+names of the C<In> classes are the official Unicode script and block
+names but with all non-alphanumeric characters removed, for example
+the block name C<"Latin-1 Supplement"> becomes C<\p{InLatin1Supplement}>.
+
+Here is the list as of Unicode 3.1.0 (the two-letter classes) and
+as defined by Perl (the one-letter classes) (in Unicode materials
+what Perl calls C<L> is often called C<L&>):
+
+ L Letter
+ Lu Letter, Uppercase
+ Ll Letter, Lowercase
+ Lt Letter, Titlecase
+ Lm Letter, Modifier
+ Lo Letter, Other
+ M Mark
+ Mn Mark, Non-Spacing
+ Mc Mark, Spacing Combining
+ Me Mark, Enclosing
+ N Number
+ Nd Number, Decimal Digit
+ Nl Number, Letter
+ No Number, Other
+ P Punctuation
+ Pc Punctuation, Connector
+ Pd Punctuation, Dash
+ Ps Punctuation, Open
+ Pe Punctuation, Close
+ Pi Punctuation, Initial quote
+ (may behave like Ps or Pe depending on usage)
+ Pf Punctuation, Final quote
+ (may behave like Ps or Pe depending on usage)
+ Po Punctuation, Other
+ S Symbol
+ Sm Symbol, Math
+ Sc Symbol, Currency
+ Sk Symbol, Modifier
+ So Symbol, Other
+ Z Separator
+ Zs Separator, Space
+ Zl Separator, Line
+ Zp Separator, Paragraph
+ C Other
+ Cc Other, Control
+ Cf Other, Format
+ Cs Other, Surrogate
+ Co Other, Private Use
+ Cn Other, Not Assigned (Unicode defines no Cn characters)
+
+Additionally, because scripts differ in their directionality
+(for example Hebrew is written right to left), all characters
+have their directionality defined:
+
+ BidiL Left-to-Right
+ BidiLRE Left-to-Right Embedding
+ BidiLRO Left-to-Right Override
+ BidiR Right-to-Left
+ BidiAL Right-to-Left Arabic
+ BidiRLE Right-to-Left Embedding
+ BidiRLO Right-to-Left Override
+ BidiPDF Pop Directional Format
+ BidiEN European Number
+ BidiES European Number Separator
+ BidiET European Number Terminator
+ BidiAN Arabic Number
+ BidiCS Common Number Separator
+ BidiNSM Non-Spacing Mark
+ BidiBN Boundary Neutral
+ BidiB Paragraph Separator
+ BidiS Segment Separator
+ BidiWS Whitespace
+ BidiON Other Neutrals
+
+=head2 Scripts
+
+The scripts available for C<\p{In...}> and C<\P{In...}>, for example
+\p{InCyrillic>, are as follows, for example C<\p{InLatin}> or C<\P{InHan}>:
+
+ Latin
+ Greek
+ Cyrillic
+ Armenian
+ Hebrew
+ Arabic
+ Syriac
+ Thaana
+ Devanagari
+ Bengali
+ Gurmukhi
+ Gujarati
+ Oriya
+ Tamil
+ Telugu
+ Kannada
+ Malayalam
+ Sinhala
+ Thai
+ Lao
+ Tibetan
+ Myanmar
+ Georgian
+ Hangul
+ Ethiopic
+ Cherokee
+ CanadianAboriginal
+ Ogham
+ Runic
+ Khmer
+ Mongolian
+ Hiragana
+ Katakana
+ Bopomofo
+ Han
+ Yi
+ OldItalic
+ Gothic
+ Deseret
+ Inherited
+
+=head2 Blocks
+
+In addition to B<scripts>, Unicode also defines B<blocks> of
+characters. The difference between scripts and blocks is that the
+former concept is closer to natural languages, while the latter
+concept is more an artificial grouping based on groups of 256 Unicode
+characters. For example, the C<Latin> script contains letters from
+many blocks, but it does not contain all the characters from those
+blocks, it does not for example contain digits.
+
+For more about scripts see the UTR #24:
+http://www.unicode.org/unicode/reports/tr24/
+For more about blocks see
+http://www.unicode.org/Public/UNIDATA/Blocks.txt
+
+Because there are overlaps in naming (there are, for example, both
+a script called C<Katakana> and a block called C<Katakana>, the block
+version has C<Block> appended to its name, C<\p{InKatakanaBlock}>.
+
+Notice that this definition was introduced in Perl 5.8.0: in Perl
+5.6.0 only the blocks were used; in Perl 5.8.0 scripts became the
+preferential character class definition; this meant that the
+definitions of some character classes changed (the ones in the
+below list that have the C<Block> appended).
+
+ BasicLatin
+ Latin1Supplement
+ LatinExtendedA
+ LatinExtendedB
+ IPAExtensions
+ SpacingModifierLetters
+ CombiningDiacriticalMarks
+ GreekBlock
+ CyrillicBlock
+ ArmenianBlock
+ HebrewBlock
+ ArabicBlock
+ SyriacBlock
+ ThaanaBlock
+ DevanagariBlock
+ BengaliBlock
+ GurmukhiBlock
+ GujaratiBlock
+ OriyaBlock
+ TamilBlock
+ TeluguBlock
+ KannadaBlock
+ MalayalamBlock
+ SinhalaBlock
+ ThaiBlock
+ LaoBlock
+ TibetanBlock
+ MyanmarBlock
+ GeorgianBlock
+ HangulJamo
+ EthiopicBlock
+ CherokeeBlock
+ UnifiedCanadianAboriginalSyllabics
+ OghamBlock
+ RunicBlock
+ KhmerBlock
+ MongolianBlock
+ LatinExtendedAdditional
+ GreekExtended
+ GeneralPunctuation
+ SuperscriptsandSubscripts
+ CurrencySymbols
+ CombiningMarksforSymbols
+ LetterlikeSymbols
+ NumberForms
+ Arrows
+ MathematicalOperators
+ MiscellaneousTechnical
+ ControlPictures
+ OpticalCharacterRecognition
+ EnclosedAlphanumerics
+ BoxDrawing
+ BlockElements
+ GeometricShapes
+ MiscellaneousSymbols
+ Dingbats
+ BraillePatterns
+ CJKRadicalsSupplement
+ KangxiRadicals
+ IdeographicDescriptionCharacters
+ CJKSymbolsandPunctuation
+ HiraganaBlock
+ KatakanaBlock
+ BopomofoBlock
+ HangulCompatibilityJamo
+ Kanbun
+ BopomofoExtended
+ EnclosedCJKLettersandMonths
+ CJKCompatibility
+ CJKUnifiedIdeographsExtensionA
+ CJKUnifiedIdeographs
+ YiSyllables
+ YiRadicals
+ HangulSyllables
+ HighSurrogates
+ HighPrivateUseSurrogates
+ LowSurrogates
+ PrivateUse
+ CJKCompatibilityIdeographs
+ AlphabeticPresentationForms
+ ArabicPresentationFormsA
+ CombiningHalfMarks
+ CJKCompatibilityForms
+ SmallFormVariants
+ ArabicPresentationFormsB
+ Specials
+ HalfwidthandFullwidthForms
+ OldItalicBlock
+ GothicBlock
+ DeseretBlock
+ ByzantineMusicalSymbols
+ MusicalSymbols
+ MathematicalAlphanumericSymbols
+ CJKUnifiedIdeographsExtensionB
+ CJKCompatibilityIdeographsSupplement
+ Tags
=item *
characters that apply to the base character. It is equivalent to
C<(?:\PM\pM*)>.
-C<use utf8> is needed to enable this. See above.
-
=item *
-The C<tr///> operator translates characters instead of bytes. It can also
-be forced to translate between 8-bit codes and UTF-8 regardless of the
-surrounding utf8 state. For instance, if you know your input in Latin-1,
-you can say:
-
- use utf8;
- while (<>) {
- tr/\0-\xff//CU; # latin1 char to utf8
- ...
- }
-
-Similarly you could translate your output with
-
- tr/\0-\x{ff}//UC; # utf8 to latin1 char
-
-No, C<s///> doesn't take /U or /C (yet?).
-
-C<use utf8> is needed to enable this. See above.
+The C<tr///> operator translates characters instead of bytes. Note
+that the C<tr///CU> functionality has been removed, as the interface
+was a mistake. For similar functionality see pack('U0', ...) and
+pack('C0', ...).
=item *
=item *
Most operators that deal with positions or lengths in the string will
-automatically switch to using character positions, including C<chop()>,
-C<substr()>, C<pos()>, C<index()>, C<rindex()>, C<sprintf()>,
-C<write()>, and C<length()>. Operators that specifically don't switch
-include C<vec()>, C<pack()>, and C<unpack()>. Operators that really
-don't care include C<chomp()>, as well as any other operator that
-treats a string as a bucket of bits, such as C<sort()>, and the
-operators dealing with filenames.
+automatically switch to using character positions, including
+C<chop()>, C<substr()>, C<pos()>, C<index()>, C<rindex()>,
+C<sprintf()>, C<write()>, and C<length()>. Operators that
+specifically don't switch include C<vec()>, C<pack()>, and
+C<unpack()>. Operators that really don't care include C<chomp()>, as
+well as any other operator that treats a string as a bucket of bits,
+such as C<sort()>, and the operators dealing with filenames.
=item *
=item *
+The bit string operators C<& | ^ ~> can operate on character data.
+However, for backward compatibility reasons (bit string operations
+when the characters all are less than 256 in ordinal value) one should
+not mix C<~> (the bit complement) and characters both less than 256 and
+equal or greater than 256. Most importantly, the DeMorgan's laws
+(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold.
+Another way to look at this is that the complement cannot return
+B<both> the 8-bit (byte) wide bit complement B<and> the full character
+wide bit complement.
+
+=item *
+
And finally, C<scalar reverse()> reverses by character rather than by byte.
=back
+=head2 Character encodings for input and output
+
+See L<Encode>.
+
=head1 CAVEATS
As of yet, there is no method for automatically coercing input and
-output to some encoding other than UTF-8. This is planned in the near
-future, however.
+output to some encoding other than UTF-8 or UTF-EBCDIC. This is planned
+in the near future, however.
-Whether a piece of data will be treated as "characters" or "bytes"
-by internal operations cannot be divined at the current time.
+Whether an arbitrary piece of data will be treated as "characters" or
+"bytes" by internal operations cannot be divined at the current time.
Use of locales with utf8 may lead to odd results. Currently there is
some attempt to apply 8-bit locale info to characters in the range
=head1 SEE ALSO
-L<byte>, L<utf8>, L<perlvar/"$^U">
+L<bytes>, L<utf8>, L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
=cut
projects / p5sagit/p5-mst-13.2.git / blobdiff