X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperluniintro.pod;h=86360d4a726753bd72b551eddac85f8ed4e2ec1a;hb=98af1e142028dcf116f32636ea54f4c3e9494651;hp=dcfb11bdaf79282b6cbeedf12330185335f5b312;hpb=a365f2ce4defc0d7fecd4e9484f8f958454c9192;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perluniintro.pod b/pod/perluniintro.pod index dcfb11b..86360d4 100644 --- a/pod/perluniintro.pod +++ b/pod/perluniintro.pod @@ -160,14 +160,14 @@ strings contain a character beyond 0x00FF. For example, - perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"' + perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"' produces a fairly useless mixture of native bytes and UTF-8, as well as a warning: Wide character in print at ... -To output UTF-8, use the C<:utf8> output layer. Prepending +To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending binmode(STDOUT, ":utf8"); @@ -252,7 +252,7 @@ to be interpreted as the UTF-8 encoding of Unicode characters: my $chars = pack("U0W*", 0x80, 0x42); -Likewise, you can stop such UTF-8 interpretation by using the special +Likewise, you can stop such UTF-8 interpretation by using the special C<"C0"> prefix. =head2 Handling Unicode @@ -278,7 +278,7 @@ encodings, I/O, and certain special cases: When you combine legacy data and Unicode the legacy data needs to be upgraded to Unicode. Normally ISO 8859-1 (or EBCDIC, if -applicable) is assumed. +applicable) is assumed. The C module knows about many encodings and has interfaces for doing conversions between those encodings: @@ -317,7 +317,9 @@ and on already open streams, use C: The matching of encoding names is loose: case does not matter, and many encodings have several aliases. Note that the C<:utf8> layer must always be specified exactly like that; it is I subject to -the loose matching of encoding names. +the loose matching of encoding names. Also note that C<:utf8> is unsafe for +input, because it accepts the data without validating that it is indeed valid +UTF8. See L for the C<:utf8> layer, L and L for the C<:encoding()> layer, and @@ -329,7 +331,7 @@ Unicode or legacy encodings does not magically turn the data into Unicode in Perl's eyes. To do that, specify the appropriate layer when opening files - open(my $fh,'<:utf8', 'anything'); + open(my $fh,'<:encoding(utf8)', 'anything'); my $line_of_unicode = <$fh>; open(my $fh,'<:encoding(Big5)', 'anything'); @@ -338,7 +340,7 @@ layer when opening files The I/O layers can also be specified more flexibly with the C pragma. See L, or look at the following example. - use open ':utf8'; # input and output default layer will be UTF-8 + use open ':encoding(utf8)'; # input/output default encoding will be UTF-8 open X, ">file"; print X chr(0x100), "\n"; close X; @@ -358,11 +360,6 @@ With the C pragma you can use the C<:locale> layer printf "%#x\n", ord(), "\n"; # this should print 0xc1 close I; -or you can also use the C<':encoding(...)'> layer - - open(my $epic,'<:encoding(iso-8859-7)','iliad.greek'); - my $line_of_unicode = <$epic>; - These methods install a transparent filter on the I/O stream that converts data from the specified encoding when it is read in from the stream. The result is always Unicode. @@ -411,13 +408,13 @@ by repeatedly encoding the data: local $/; ## read in the whole file of 8-bit characters $t = ; close F; - open F, ">:utf8", "file"; + open F, ">:encoding(utf8)", "file"; print F $t; ## convert to UTF-8 on output close F; If you run this code twice, the contents of the F will be twice -UTF-8 encoded. A C would have avoided the bug, or -explicitly opening also the F for input as UTF-8. +UTF-8 encoded. A C would have avoided the +bug, or explicitly opening also the F for input as UTF-8. B: the C<:utf8> and C<:encoding> features work only if your Perl has been built with the new PerlIO feature (which is the default @@ -517,8 +514,8 @@ CAPITAL LETTER As should be considered equal, or even As of any case. The long answer is that you need to consider character normalization and casing issues: see L, Unicode Technical Reports #15 and #21, I and I, http://www.unicode.org/unicode/reports/tr15/ and -http://www.unicode.org/unicode/reports/tr21/ +Mappings>, http://www.unicode.org/unicode/reports/tr15/ and +http://www.unicode.org/unicode/reports/tr21/ As of Perl 5.8.0, the "Full" case-folding of I is implemented. @@ -656,8 +653,8 @@ Use the C package to try converting it. For example, use Encode 'decode_utf8'; - eval { decode_utf8($string, Encode::FB_CROAK) }; - if ($@) { + + if (eval { decode_utf8($string, Encode::FB_CROAK); 1 }) { # $string is valid utf8 } else { # $string is not valid utf8 @@ -668,10 +665,10 @@ Or use C to try decoding it: use warnings; @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8); -If invalid, a C warning is produced. The "C0" means -"process the string character per character". Without that, the -C would work in C mode (the default if the format -string starts with C) and it would return the bytes making up the UTF-8 +If invalid, a C warning is produced. The "C0" means +"process the string character per character". Without that, the +C would work in C mode (the default if the format +string starts with C) and it would return the bytes making up the UTF-8 encoding of the target string, something that will always work. =item * @@ -726,7 +723,7 @@ but Perl doesn't know it yet, you can make Perl a believer, too: or: $Unicode = pack("U0a*", $bytes); - + You can convert well-formed UTF-8 to a sequence of bytes, but if you just want to convert random binary data into UTF-8, you can't. B. You can @@ -790,44 +787,44 @@ show a decimal number in hexadecimal. If you have just the Unicode Consortium - http://www.unicode.org/ +http://www.unicode.org/ =item * Unicode FAQ - http://www.unicode.org/unicode/faq/ +http://www.unicode.org/unicode/faq/ =item * Unicode Glossary - http://www.unicode.org/glossary/ +http://www.unicode.org/glossary/ =item * Unicode Useful Resources - http://www.unicode.org/unicode/onlinedat/resources.html +http://www.unicode.org/unicode/onlinedat/resources.html =item * Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications - http://www.alanwood.net/unicode/ +http://www.alanwood.net/unicode/ =item * UTF-8 and Unicode FAQ for Unix/Linux - http://www.cl.cam.ac.uk/~mgk25/unicode.html +http://www.cl.cam.ac.uk/~mgk25/unicode.html =item * Legacy Character Sets - http://www.czyborra.com/ - http://www.eki.ee/letter/ +http://www.czyborra.com/ +http://www.eki.ee/letter/ =item * @@ -836,7 +833,7 @@ directory $Config{installprivlib}/unicore -in Perl 5.8.0 or newer, and +in Perl 5.8.0 or newer, and $Config{installprivlib}/unicode