X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=lib%2Futf8.pm;h=ac73aa180665c706d72b2150a924e56d7694a5bb;hb=50109ad0d28b27abe5ee82def070e14b4526321c;hp=5ff900d8f930ad0b93c1b50b6f59361ab0d2a9f4;hpb=2fa62f6675d9a740da5d4e8530030bb9dd10e689;p=p5sagit%2Fp5-mst-13.2.git diff --git a/lib/utf8.pm b/lib/utf8.pm index 5ff900d..ac73aa1 100644 --- a/lib/utf8.pm +++ b/lib/utf8.pm @@ -50,22 +50,18 @@ program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based platforms). The C pragma tells Perl to switch back to treating the source text as literal bytes in the current lexical scope. -This pragma is primarily a compatibility device. Perl versions -earlier than 5.6 allowed arbitrary bytes in source code, whereas -in future we would like to standardize on the UTF-8 encoding for -source text. - B The utility functions described below are -useful for their own purposes, but they are not really part of the -"pragmatic" effect. +directly usable without C. + +Because it is not possible to reliably tell UTF-8 from native 8 bit +encodings, you need either a Byte Order Mark at the beginning of your +source code, or C, to instruct perl. -Until UTF-8 becomes the default format for source text, either this -pragma or the L pragma should be used to recognize UTF-8 -in the source. When UTF-8 becomes the standard source format, this -pragma will effectively become a no-op. For convenience in what -follows the term I is used to refer to UTF-8 on ASCII and ISO -Latin based platforms and UTF-EBCDIC on EBCDIC based platforms. +When UTF-8 becomes the standard source format, this pragma will +effectively become a no-op. For convenience in what follows the term +I is used to refer to UTF-8 on ASCII and ISO Latin based +platforms and UTF-EBCDIC on EBCDIC based platforms. See also the effects of the C<-C> switch and its cousin, the C<$ENV{PERL_UNICODE}>, in L. @@ -93,21 +89,6 @@ UTF-X. If you want to have such bytes under C, you can disable this pragma until the end the block (or file, if at top level) by C. -If you want to automatically upgrade your 8-bit legacy bytes to Unicode, -use the L pragma instead of this pragma. For example, if -you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to Unicode -as used in e.g. C and C<\x{...}>, try this: - - use encoding "latin-1"; - my $c = chr(0xc4); - my $x = "\x{c5}"; - -In case you are wondering: C is mostly the same as -C, except that C marks all string literals in the -source code as Unicode, regardless of whether they contain any high-bit bytes. -Moreover, C installs IO layers on C and C to work -with Unicode strings; see L for details. - =head2 Utility functions The following functions are defined in the C package by the @@ -118,64 +99,63 @@ you should not say that unless you really want to have UTF-8 source code. =item * $num_octets = utf8::upgrade($string) -Converts in-place the octet sequence in the native encoding +Converts in-place the internal octet sequence in the native encoding (Latin-1 or EBCDIC) to the equivalent character sequence in I. -I<$string> already encoded as characters does no harm. -Returns the number of octets necessary to represent the string as I. -Can be used to make sure that the UTF-8 flag is on, -so that C<\w> or C work as Unicode on strings -containing characters in the range 0x80-0xFF (on ASCII and -derivatives). +I<$string> already encoded as characters does no harm. Returns the +number of octets necessary to represent the string as I. Can be +used to make sure that the UTF-8 flag is on, so that C<\w> or C +work as Unicode on strings containing characters in the range 0x80-0xFF +(on ASCII and derivatives). B -Therefore I is recommended for the general purposes. - -Affected by the encoding pragma. +Therefore Encode is recommended for the general purposes; see also +L. =item * $success = utf8::downgrade($string[, FAIL_OK]) -Converts in-place the character sequence in I -to the equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). -I<$string> already encoded as octets does no harm. -Returns true on success. On failure dies or, if the value of -C is true, returns false. -Can be used to make sure that the UTF-8 flag is off, -e.g. when you want to make sure that the substr() or length() function -works with the usually faster byte algorithm. +Converts in-place the internal octet sequence in I to the +equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). +I<$string> already encoded as native 8 bit does no harm. Can be used to +make sure that the UTF-8 flag is off, e.g. when you want to make sure +that the substr() or length() function works with the usually faster +byte algorithm. -B -Therefore I is recommended for the general purposes. +Fails if the original I sequence cannot be represented in the +native 8 bit encoding. On failure dies or, if the value of C is +true, returns false. -B affected by the encoding pragma. +Returns true on success. -B this function is experimental and may change -or be removed without notice. +B +Therefore Encode is recommended for the general purposes; see also +L. =item * utf8::encode($string) -Converts in-place the character sequence to the corresponding octet sequence -in I. The UTF-8 flag is turned off. Returns nothing. +Converts in-place the character sequence to the corresponding octet +sequence in I. The UTF8 flag is turned off, so that after this +operation, the string is a byte string. Returns nothing. B -Therefore I is recommended for the general purposes. +Therefore Encode is recommended for the general purposes; see also +L. -=item * utf8::decode($string) +=item * $success = utf8::decode($string) -Attempts to convert in-place the octet sequence in I -to the corresponding character sequence. The UTF-8 flag is turned on -only if the source string contains multiple-byte I characters. -If I<$string> is invalid as I, returns false; otherwise returns true. +Attempts to convert in-place the octet sequence in I to the +corresponding character sequence. The UTF-8 flag is turned on only if +the source string contains multiple-byte I characters. If +I<$string> is invalid as I, returns false; otherwise returns +true. B -Therefore I is recommended for the general purposes. - -B this function is experimental and may change -or be removed without notice. +Therefore Encode is recommended for the general purposes; see also +L. =item * $flag = utf8::is_utf8(STRING) -(Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally -the same as Encode::is_utf8(). +(Since Perl 5.8.1) Test whether STRING is in UTF-8 internally. +Functionally the same as Encode::is_utf8(). =item * $flag = utf8::valid(STRING) @@ -193,10 +173,10 @@ cleared. See L for more on the UTF8 flag and the C API functions C, C, C, and C, which are wrapped by the Perl functions C, C, C and -C. Note that in the Perl 5.8.0 and 5.8.1 implementation -the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode, -utf8::upgrade, and utf8::downgrade are always available, without a -C statement-- this may change in future releases. +C. Also, the functions utf8::is_utf8, utf8::valid, +utf8::encode, utf8::decode, utf8::upgrade, and utf8::downgrade are +actually internal, and thus always available, without a C +statement. =head1 BUGS @@ -213,6 +193,6 @@ portable answers. =head1 SEE ALSO -L, L, L, L, L +L, L, L, L, L =cut