X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=lib%2Futf8.pm;h=a985021f32a40f2857b6eb4d5e0a59decbc7b376;hb=a06bfbf46ad8e21060b59ed409ba2f87fbfcdc35;hp=e0b9f85b26397a7642df58d49b9720c1bc74905a;hpb=bd7017d37d1ea75ed228b98d66032e165497283f;p=p5sagit%2Fp5-mst-13.2.git diff --git a/lib/utf8.pm b/lib/utf8.pm index e0b9f85..a985021 100644 --- a/lib/utf8.pm +++ b/lib/utf8.pm @@ -2,7 +2,7 @@ package utf8; $utf8::hint_bits = 0x00800000; -our $VERSION = '1.04'; +our $VERSION = '1.07'; sub import { $^H |= $utf8::hint_bits; @@ -50,22 +50,18 @@ program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based platforms). The C pragma tells Perl to switch back to treating the source text as literal bytes in the current lexical scope. -This pragma is primarily a compatibility device. Perl versions -earlier than 5.6 allowed arbitrary bytes in source code, whereas -in future we would like to standardize on the UTF-8 encoding for -source text. - B The utility functions described below are -useful for their own purposes, but they are not really part of the -"pragmatic" effect. +directly usable without C. + +Because it is not possible to reliably tell UTF-8 from native 8 bit +encodings, you need either a Byte Order Mark at the beginning of your +source code, or C, to instruct perl. -Until UTF-8 becomes the default format for source text, either this -pragma or the L pragma should be used to recognize UTF-8 -in the source. When UTF-8 becomes the standard source format, this -pragma will effectively become a no-op. For convenience in what -follows the term I is used to refer to UTF-8 on ASCII and ISO -Latin based platforms and UTF-EBCDIC on EBCDIC based platforms. +When UTF-8 becomes the standard source format, this pragma will +effectively become a no-op. For convenience in what follows the term +I is used to refer to UTF-8 on ASCII and ISO Latin based +platforms and UTF-EBCDIC on EBCDIC based platforms. See also the effects of the C<-C> switch and its cousin, the C<$ENV{PERL_UNICODE}>, in L. @@ -77,7 +73,7 @@ Enabling the C pragma has the following effect: =item * Bytes in the source text that have their high-bit set will be treated -as being part of a literal UTF-8 character. This includes most +as being part of a literal UTF-X sequence. This includes most literals such as identifier names, string constants, and constant regular expression patterns. @@ -89,20 +85,9 @@ treated as being part of a literal UTF-EBCDIC character. Note that if you have bytes with the eighth bit on in your script (for example embedded Latin-1 in your string literals), C will be unhappy since the bytes are most probably not well-formed -UTF-8. If you want to have such bytes and use utf8, you can disable -utf8 until the end the block (or file, if at top level) by C. - -If you want to automatically upgrade your 8-bit legacy bytes to UTF-8, -use the L pragma instead of this pragma. For example, if -you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to UTF-8 -as used in e.g. C and C<\x{...}>, try this: - - use encoding "latin-1"; - my $c = chr(0xc4); - my $x = "\x{c5}"; - -In case you are wondering: yes, C works much -the same as C. +UTF-X. If you want to have such bytes under C, you can disable +this pragma until the end the block (or file, if at top level) by +C. =head2 Utility functions @@ -114,64 +99,63 @@ you should not say that unless you really want to have UTF-8 source code. =item * $num_octets = utf8::upgrade($string) -Converts in-place the octet sequence in the native encoding +Converts in-place the internal octet sequence in the native encoding (Latin-1 or EBCDIC) to the equivalent character sequence in I. -I<$string> already encoded as characters does no harm. -Returns the number of octets necessary to represent the string as I. -Can be used to make sure that the UTF-8 flag is on, -so that C<\w> or C work as Unicode on strings -containing characters in the range 0x80-0xFF (on ASCII and -derivatives). +I<$string> already encoded as characters does no harm. Returns the +number of octets necessary to represent the string as I. Can be +used to make sure that the UTF-8 flag is on, so that C<\w> or C +work as Unicode on strings containing characters in the range 0x80-0xFF +(on ASCII and derivatives). B -Therefore I is recommended for the general purposes. - -Affected by the encoding pragma. +Therefore Encode is recommended for the general purposes; see also +L. =item * $success = utf8::downgrade($string[, FAIL_OK]) -Converts in-place the character sequence in I -to the equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). -I<$string> already encoded as octets does no harm. -Returns true on success. On failure dies or, if the value of -C is true, returns false. -Can be used to make sure that the UTF-8 flag is off, -e.g. when you want to make sure that the substr() or length() function -works with the usually faster byte algorithm. +Converts in-place the internal octet sequence in I to the +equivalent octet sequence in the native encoding (Latin-1 or EBCDIC). +I<$string> already encoded as native 8 bit does no harm. Can be used to +make sure that the UTF-8 flag is off, e.g. when you want to make sure +that the substr() or length() function works with the usually faster +byte algorithm. -B -Therefore I is recommended for the general purposes. +Fails if the original I sequence cannot be represented in the +native 8 bit encoding. On failure dies or, if the value of C is +true, returns false. -B affected by the encoding pragma. +Returns true on success. -B this function is experimental and may change -or be removed without notice. +B +Therefore Encode is recommended for the general purposes; see also +L. =item * utf8::encode($string) -Converts in-place the character sequence to the corresponding octet sequence -in I. The UTF-8 flag is turned off. Returns nothing. +Converts in-place the character sequence to the corresponding octet +sequence in I. The UTF8 flag is turned off, so that after this +operation, the string is a byte string. Returns nothing. B -Therefore I is recommended for the general purposes. +Therefore Encode is recommended for the general purposes; see also +L. -=item * utf8::decode($string) +=item * $success = utf8::decode($string) -Attempts to convert in-place the octet sequence in I -to the corresponding character sequence. The UTF-8 flag is turned on -only if the source string contains multiple-byte I characters. -If I<$string> is invalid as I, returns false; otherwise returns true. +Attempts to convert in-place the octet sequence in I to the +corresponding character sequence. The UTF-8 flag is turned on only if +the source string contains multiple-byte I characters. If +I<$string> is invalid as I, returns false; otherwise returns +true. B -Therefore I is recommended for the general purposes. - -B this function is experimental and may change -or be removed without notice. +Therefore Encode is recommended for the general purposes; see also +L. =item * $flag = utf8::is_utf8(STRING) -(Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally -the same as Encode::is_utf8(). +(Since Perl 5.8.1) Test whether STRING is in UTF-8 internally. +Functionally the same as Encode::is_utf8(). =item * $flag = utf8::valid(STRING) @@ -189,10 +173,10 @@ cleared. See L for more on the UTF8 flag and the C API functions C, C, C, and C, which are wrapped by the Perl functions C, C, C and -C. Note that in the Perl 5.8.0 and 5.8.1 implementation -the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode, -utf8::upgrade, and utf8::downgrade are always available, without a -C statement-- this may change in future releases. +C. Also, the functions utf8::is_utf8, utf8::valid, +utf8::encode, utf8::decode, utf8::upgrade, and utf8::downgrade are +actually internal, and thus always available, without a C +statement. =head1 BUGS @@ -209,6 +193,6 @@ portable answers. =head1 SEE ALSO -L, L, L, L, L +L, L, L, L, L =cut