From: Jarkko Hietaniemi Date: Sun, 12 Aug 2001 14:16:44 +0000 (+0000) Subject: Dispell the "use utf8" superstition. X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=commitdiff_plain;h=ad0029c435199eaf70c06265f639c1af50f36906;p=p5sagit%2Fp5-mst-13.2.git Dispell the "use utf8" superstition. p4raw-id: //depot/perl@11643 --- diff --git a/lib/utf8.pm b/lib/utf8.pm index 0885e67..cb19686 100644 --- a/lib/utf8.pm +++ b/lib/utf8.pm @@ -1,6 +1,5 @@ package utf8; - $utf8::hint_bits = 0x00800000; our $VERSION = '1.00'; @@ -34,9 +33,6 @@ utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code =head1 DESCRIPTION -WARNING: The implementation of Unicode support in Perl is incomplete. -See L for the exact details. - The C pragma tells the Perl parser to allow UTF-8 in the program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based platforms). The C pragma tells Perl to switch back to treating @@ -49,33 +45,21 @@ source text. Until UTF-8 becomes the default format for source text, this pragma should be used to recognize UTF-8 in the source. When UTF-8 becomes the standard source format, this pragma will effectively become a no-op. For convenience in what follows the -term UTF-X is used to refer to UTF-8 on ASCII and ISO Latin based +term I is used to refer to UTF-8 on ASCII and ISO Latin based platforms and UTF-EBCDIC on EBCDIC based platforms. -Enabling the C pragma has the following effects: +Enabling the C pragma has the following effect: =over 4 =item * Bytes in the source text that have their high-bit set will be treated -as being part of a literal UTF-8 character. This includes most literals -such as identifiers, string constants, constant regular expression patterns -and package names. On EBCDIC platforms characters in the Latin 1 -character set are treated as being part of a literal UTF-EBCDIC character. - -=item * - -In the absence of inputs marked as UTF-X, regular expressions within the -scope of this pragma will default to using character semantics instead -of byte semantics. - - @bytes_or_chars = split //, $data; # may split to bytes if data - # $data isn't UTF-X - { - use utf8; # force char semantics - @chars = split //, $data; # splits characters - } +as being part of a literal UTF-8 character. This includes most +literals such as identifiers, string constants, constant regular +expression patterns and package names. On EBCDIC platforms characters +in the Latin 1 character set are treated as being part of a literal +UTF-EBCDIC character. =back @@ -87,8 +71,9 @@ The following functions are defined in the C package by the perl core. =item * $num_octets = utf8::upgrade($string); -Converts internal representation of string to the perls internal UTF-X form. -Returns the number of octets necessary to represent the string as UTF-X. +Converts internal representation of string to the Perl's internal +I form. Returns the number of octets necessary to represent +the string as I. =item * utf8::downgrade($string[, CHECK]) @@ -97,11 +82,12 @@ Converts internal representation of string to be un-encoded bytes. =item * utf8::encode($string) Converts (in-place) I<$string> from logical characters to octet sequence -representing it in perl's UTF-X encoding. +representing it in Perl's I encoding. =item * $flag = utf8::decode($string) -Attempts to convert I<$string> in-place from perl's UTF-X encoding into logical characters. +Attempts to convert I<$string> in-place from Perl's I encoding +into logical characters. =back diff --git a/pod/perlfunc.pod b/pod/perlfunc.pod index 4a76999..56937f4 100644 --- a/pod/perlfunc.pod +++ b/pod/perlfunc.pod @@ -682,9 +682,9 @@ On POSIX systems, you can detect this condition this way: Returns the character represented by that NUMBER in the character set. For example, C is C<"A"> in either ASCII or Unicode, and -chr(0x263a) is a Unicode smiley face. Within the scope of C, -characters higher than 127 are encoded in Unicode; if you don't want -this, temporarily C or use C +chr(0x263a) is a Unicode smiley face. Note that characters from +127 to 255 (inclusive) are not encoded in Unicode for backward +compatibility reasons. For the reverse, use L. See L for more about Unicode. @@ -2310,9 +2310,9 @@ C work. =item lc Returns an lowercased version of EXPR. This is the internal function -implementing the C<\L> escape in double-quoted strings. -Respects current LC_CTYPE locale if C in force. See L -and L. +implementing the C<\L> escape in double-quoted strings. Respects +current LC_CTYPE locale if C in force. See L +and L. If EXPR is omitted, uses C<$_>. @@ -2320,9 +2320,10 @@ If EXPR is omitted, uses C<$_>. =item lcfirst -Returns the value of EXPR with the first character lowercased. This is -the internal function implementing the C<\l> escape in double-quoted strings. -Respects current LC_CTYPE locale if C in force. See L. +Returns the value of EXPR with the first character lowercased. This +is the internal function implementing the C<\l> escape in +double-quoted strings. Respects current LC_CTYPE locale if C in force. See L and L. If EXPR is omitted, uses C<$_>. @@ -3080,8 +3081,8 @@ follows: P A pointer to a structure (fixed-length string). u A uuencoded string. - U A Unicode character number. Encodes to UTF-8 internally. - Works even if C is not in effect. + U A Unicode character number. Encodes to UTF-8 internally + (or UTF-EBCDIC in EBCDIC platforms). w A BER compressed integer. Its bytes represent an unsigned integer in base 128, most significant digit first, with as @@ -5367,10 +5368,11 @@ otherwise. =item uc Returns an uppercased version of EXPR. This is the internal function -implementing the C<\U> escape in double-quoted strings. -Respects current LC_CTYPE locale if C in force. See L. -Under Unicode (C) it uses the standard Unicode uppercase mappings. (It -does not attempt to do titlecase mapping on initial letters. See C for that.) +implementing the C<\U> escape in double-quoted strings. Respects +current LC_CTYPE locale if C in force. See L +and L. Under Unicode it uses the standard Unicode +uppercase mappings. (It does not attempt to do titlecase mapping on +initial letters. See C for that.) If EXPR is omitted, uses C<$_>. @@ -5378,11 +5380,10 @@ If EXPR is omitted, uses C<$_>. =item ucfirst -Returns the value of EXPR with the first character -in uppercase (titlecase in Unicode). This is -the internal function implementing the C<\u> escape in double-quoted strings. -Respects current LC_CTYPE locale if C in force. See L -and L. +Returns the value of EXPR with the first character in uppercase +(titlecase in Unicode). This is the internal function implementing +the C<\u> escape in double-quoted strings. Respects current LC_CTYPE +locale if C in force. See L and L. If EXPR is omitted, uses C<$_>. diff --git a/pod/perlretut.pod b/pod/perlretut.pod index acb95cd..95e3f03 100644 --- a/pod/perlretut.pod +++ b/pod/perlretut.pod @@ -1647,13 +1647,18 @@ sequence of bytes (the old way) or as a sequence of Unicode characters than C may be represented using the C<\x{hex}> notation, with C a hexadecimal integer: - use utf8; # We will be doing Unicode processing /\x{263a}/; # match a Unicode smiley face :) Unicode characters in the range of 128-255 use two hexadecimal digits with braces: C<\x{ab}>. Note that this is different than C<\xab>, -which is just a hexadecimal byte with no Unicode -significance. +which is just a hexadecimal byte with no Unicode significance. + +B: in perl 5.6.0 it used to be that one needed to say C +to use any Unicode features. This is no more the case: for almost all +Unicode processing, the explicit C pragma is not needed. +(The only case where it matters is if your Perl script is in Unicode, +that is, encoded in UTF-8/UTF-16/UTF-EBCDIC: then an explicit C +is needed.) Figuring out the hexadecimal sequence of a Unicode character you want or deciphering someone else's hexadecimal Unicode regexp is about as @@ -1664,15 +1669,12 @@ specified in the Unicode standard. For instance, if we wanted to represent or match the astrological sign for the planet Mercury, we could use - use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "abc\N{MERCURY}def"; $x =~ /\N{MERCURY}/; # matches One can also use short names or restrict names to a certain alphabet: - use utf8; # We will be doing Unicode processing - use charnames ':full'; print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n"; @@ -1693,7 +1695,6 @@ characters, but matching a single byte is required, we can use the C<\C> escape sequence. C<\C> is a character class akin to C<.> except that it matches I byte 0-255. So - use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "a"; $x =~ /\C/; # matches 'a', eats one byte @@ -1715,7 +1716,6 @@ the C<\P{name}> character class, which is the negation of the C<\p{name}> class. For example, to match lower and uppercase characters, - use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "BOB"; $x =~ /^\p{IsUpper}/; # matches, uppercase char class @@ -1788,7 +1788,6 @@ be used just like C<\d>, both inside and outside of character classes: /\s+[abc[:digit:]xyz]\s*/; # match a,b,c,x,y,z, or a digit /^=item\s[:digit:]/; # match '=item', # followed by a space and a digit - use utf8; use charnames ":full"; /\s+[abc\p{IsDigit}xyz]\s+/; # match a,b,c,x,y,z, or a digit /^=item\s\p{IsDigit}/; # match '=item', diff --git a/pod/perltoc.pod b/pod/perltoc.pod index a250c61..a0960bb 100644 --- a/pod/perltoc.pod +++ b/pod/perltoc.pod @@ -1992,7 +1992,7 @@ LC_NUMERIC, LC_TIME, LANG =item Important Caveats Input and Output Disciplines, Regular Expressions, C still needed -to enable a few features +to enable UTF-8/UTF-EBCDIC in scripts =item Byte and Character semantics @@ -6003,6 +6003,50 @@ Source, Compiled Module Source, Perl Modules/Scripts =back +=head2 perldos - Perl under DOS, W31, W95. + +=over 4 + +=item SYNOPSIS + +=item DESCRIPTION + +=over 4 + +=item Prerequisites for Compiling Perl on DOS + +DJGPP, Pthreads + +=item Shortcomings of Perl under DOS + +=item Building Perl on DOS + +=item Testing Perl on DOS + +=item Installation of Perl on DOS + +=back + +=item BUILDING AND INSTALLING MODULES ON DOS + +=over 4 + +=item Building Prerequisites for Perl on DOS + +=item Unpacking CPAN Modules on DOS + +=item Building Non-XS Modules on DOS + +=item Building XS Modules on DOS + +=back + +=item AUTHOR + +=item SEE ALSO + +=back + =head2 perlepoc, README.epoc - Perl for EPOC =over 4 @@ -6873,6 +6917,48 @@ LIST, waitpid PID,FLAGS =back +=head2 perlwin32 - Perl under Win32 + +=over 4 + +=item SYNOPSIS + +=item DESCRIPTION + +=over 4 + +=item Setting Up Perl on Win32 + +Make, Command Shell, Borland C++, Microsoft Visual C++, Mingw32 with GCC + +=item Building + +=item Testing Perl on Win32 + +=item Installation of Perl on Win32 + +=item Usage Hints for Perl on Win32 + +Environment Variables, File Globbing, Using perl from the command line, +Building Extensions, Command-line Wildcard Expansion, Win32 Specific +Extensions, Running Perl Scripts, Miscellaneous Things + +=back + +=item BUGS AND CAVEATS + +=item AUTHORS + +Gary Ng E71564.1743@CompuServe.COME, Gurusamy Sarathy +Egsar@activestate.comE, Nick Ing-Simmons +Enick@ing-simmons.netE + +=item SEE ALSO + +=item HISTORY + +=back + =head1 PRAGMA DOCUMENTATION =head2 attrs - set/get attributes of a subroutine (deprecated) diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index 914ce04..f429be7 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -40,16 +40,16 @@ presented with Unicode data, or a traditional byte scheme when presented with byte data. The implementation is still new and (particularly on EBCDIC platforms) may need further work. -=item C still needed to enable a few features +=item C still needed to enable UTF-8/UTF-EBCDIC in scripts The C pragma implements the tables used for Unicode support. These tables are automatically loaded on demand, so the C pragma need not normally be used. However, as a compatibility measure, this pragma must be explicitly -used to enable recognition of UTF-8 encoded literals and identifiers -in the source text on ASCII based machines or recognize UTF-EBCDIC -encoded literals and identifiers on EBCDIC based machines. +used to enable recognition of UTF-8 in the Perl scripts themselves on +ASCII based machines or recognize UTF-EBCDIC on EBCDIC based machines. +B is needed>. =back