=head1 DESCRIPTION
-=head2 Important Caveat
+=head2 Important Caveats
-WARNING: The implementation of Unicode support in Perl is incomplete.
+WARNING: While the implementation of Unicode support in Perl is now fairly
+complete it is still evolving to some extent.
-The following areas need further work.
+In particular the way Unicode is handled on EBCDIC platforms is still rather
+experimental. On such a platform references to UTF-8 encoding in this
+document and elsewhere should be read as meaning UTF-EBCDIC as specified
+in Unicode Technical Report 16 unless ASCII vs EBCDIC issues are specifically
+discussed. There is no C<utfebcdic> pragma or ":utfebcdic" layer, rather
+"utf8" and ":utf8" are re-used to mean platform's "natural" 8-bit encoding
+of Unicode. See L<perlebcdic> for more discussion of the issues.
-=over
+The following areas are still under development.
+
+=over 4
=item Input and Output Disciplines
-There is currently no easy way to mark data read from a file or other
-external source as being utf8. This will be one of the major areas of
-focus in the near future.
+A filehandle can be marked as containing perl's internal Unicode encoding
+(UTF-8 or UTF-EBCDIC) by opening it with the ":utf8" layer.
+Other encodings can be converted to perl's encoding on input, or from
+perl's encoding on output by use of the ":encoding()" layer.
+There is not yet a clean way to mark the perl source itself as being
+in an particular encoding.
=item Regular Expressions
-The existing regular expression compiler does not produce polymorphic
-opcodes. This means that the determination on whether to match Unicode
-characters is made when the pattern is compiled, based on whether the
-pattern contains Unicode characters, and not when the matching happens
-at run time. This needs to be changed to adaptively match Unicode if
-the string to be matched is Unicode.
+The regular expression compiler does now attempt to produce
+polymorphic opcodes. That is the pattern should now adapt to the data
+and automatically switch to the Unicode character scheme when presented
+with Unicode data, or a traditional byte scheme when presented with
+byte data. The implementation is still new and (particularly on
+EBCDIC platforms) may need further work.
=item C<use utf8> still needed to enable a few features
If the C<-C> command line switch is used, (or the ${^WIDE_SYSTEM_CALLS}
global flag is set to C<1>), all system calls will use the
corresponding wide character APIs. This is currently only implemented
-on Windows.
+on Windows since UNIXes lack API standard on this area.
Regardless of the above, the C<bytes> pragma can always be used to force
byte semantics in a particular lexical scope. See L<bytes>.
characters, but you can also specify a particular character with an
extension of the C<\x> notation. UTF-8 characters are specified by
putting the hexadecimal code within curlies after the C<\x>. For instance,
-a Unicode smiley face is C<\x{263A}>. A character in the Latin-1 range
-(128..255) should be written C<\x{ab}> rather than C<\xab>, since the
-former will turn into a two-byte UTF-8 code, while the latter will
-continue to be interpreted as generating a 8-bit byte rather than a
-character. In fact, if C<-w> is turned on, it will produce a warning
-that you might be generating invalid UTF-8.
+a Unicode smiley face is C<\x{263A}>.
=item *
=item *
-The C<tr///> operator translates characters instead of bytes. It can also
-be forced to translate between 8-bit codes and UTF-8. For instance, if you
-know your input in Latin-1, you can say:
-
- while (<>) {
- tr/\0-\xff//CU; # latin1 char to utf8
- ...
- }
-
-Similarly you could translate your output with
-
- tr/\0-\x{ff}//UC; # utf8 to latin1 char
-
-No, C<s///> doesn't take /U or /C (yet?).
+The C<tr///> operator translates characters instead of bytes. Note
+that the C<tr///CU> functionality has been removed, as the interface
+was a mistake. For similar functionality see pack('U0', ...) and
+pack('C0', ...).
=item *
=item *
+The bit string operators C<& | ^ ~> can operate on character data.
+However, for backward compatibility reasons (bit string operations
+when the characters all are less than 256 in ordinal value) one cannot
+mix C<~> (the bit complement) and characters both less than 256 and
+equal or greater than 256. Most importantly, the DeMorgan's laws
+(C<~($x|$y) eq ~$x&~$y>, C<~($x&$y) eq ~$x|~$y>) won't hold.
+Another way to look at this is that the complement cannot return
+B<both> the 8-bit (byte) wide bit complement, and the full character
+wide bit complement.
+
+=item *
+
And finally, C<scalar reverse()> reverses by character rather than by byte.
=back
projects / p5sagit/p5-mst-13.2.git / blobdiff