X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlunitut.pod;h=fc352d5aad175aa90592273ffc168360051605fc;hb=000c65fce914409ad42f49763dbced48187b5baf;hp=ae8d0b15662a50d2b0bdc2d986166f55f03c49ef;hpb=aadaa45530a169c37ceb95c3ed76644ccbd2a644;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlunitut.pod b/pod/perlunitut.pod index ae8d0b1..fc352d5 100644 --- a/pod/perlunitut.pod +++ b/pod/perlunitut.pod @@ -37,11 +37,13 @@ You may have to re-read this entire section a few times... =head3 Unicode B is a character set with room for lots of characters. The ordinal -value of a character is called a B. +value of a character is called a B. (But in practice, the +distinction between code point and character is blurred, so the terms often +are used interchangeably.) -There are many, many code points, but computers work with bytes, and a byte can -have only 256 values. Unicode has many more characters, so you need a method -to make these accessible. +There are many, many code points, but computers work with bytes, and a byte has +room for only 256 values. Unicode has many more characters, so you need a +method to make these accessible. Unicode is encoded using several competing encodings, of which UTF-8 is the most used. In a Unicode encoding, multiple subsequent bytes can be used to @@ -170,234 +172,15 @@ known. "Content-Type: text/plain; charset=UTF-8", "Content-Length: $byte_count" -=head2 Q and A - -=head3 This isn't really a Unicode tutorial, is it? - -No, Perl has an abstracted interface for all supported character encodings, so -this is actually a generic C tutorial. But many people think that -Unicode is special and magical, and I didn't want to disappoint them, so I -decided to call this document a Unicode tutorial. - -=head3 What about binary data, like images? - -Well, apart from a bare C, you shouldn't treat them specially. -(The binmode is needed because otherwise Perl may convert line endings on Win32 -systems.) - -Be careful, though, to never combine text strings with binary strings. If you -need text in a binary stream, encode your text strings first using the -appropriate encoding, then join them with binary strings. See also: "What if I -don't encode?". - -=head3 What about the UTF-8 flag? - -Please, unless you're hacking the internals, or debugging weirdness, don't -think about the UTF-8 flag at all. That means that you very probably shouldn't -use C, C<_utf8_on> or C<_utf8_off> at all. - -Perl's internal format happens to be UTF-8. Unfortunately, Perl can't keep a -secret, so everyone knows about this. That is the source of much confusion. -It's better to pretend that the internal format is some unknown encoding, -and that you always have to encode and decode explicitly. - -=head3 When should I decode or encode? - -Whenever you're communicating with anything that is external to your perl -process, like a database, a text file, a socket, or another program. Even if -the thing you're communicating with is also written in Perl. - -=head3 What if I don't decode? - -Whenever your encoded, binary string is used together with a text string, Perl -will assume that your binary string was encoded with ISO-8859-1, also known as -latin-1. If it wasn't latin-1, then your data is unpleasantly converted. For -example, if it was UTF-8, the individual bytes of multibyte characters are seen -as separate characters, and then again converted to UTF-8. Such double encoding -can be compared to double HTML encoding (C<&gt;>), or double URI encoding -(C<%253E>). - -This silent implicit decoding is known as "upgrading". That may sound -positive, but it's best to avoid it. - -=head3 What if I don't encode? - -Your text string will be sent using the bytes in Perl's internal format. In -some cases, Perl will warn you that you're doing something wrong, with a -friendly warning: - - Wide character in print at example.pl line 2. - -Because the internal format is often UTF-8, these bugs are hard to spot, -because UTF-8 is usually the encoding you wanted! But don't be lazy, and don't -use the fact that Perl's internal format is UTF-8 to your advantage. Encode -explicitly to avoid weird bugs, and to show to maintenance programmers that you -thought this through. - -=head3 Is there a way to automatically decode or encode? - -If all data that comes from a certain handle is encoded in exactly the same -way, you can tell the PerlIO system to automatically decode everything, with -the C layer. If you do this, you can't accidentally forget to decode -or encode anymore, on things that use the layered handle. - -You can provide this layer when Cing the file: - - open my $fh, '>:encoding(UTF-8)', $filename; # auto encoding on write - open my $fh, '<:encoding(UTF-8)', $filename; # auto decoding on read - -Or if you already have an open filehandle: - - binmode $fh, ':encoding(UTF-8)'; - -Some database drivers for DBI can also automatically encode and decode, but -that is typically limited to the UTF-8 encoding, because they cheat. - -=head3 Cheat?! Tell me, how can I cheat? - -Well, because Perl's internal format is UTF-8, you can just skip the encoding -or decoding step, and manipulate the UTF-8 flag directly. - -Instead of C<:encoding(UTF-8)>, you can simply use C<:utf8>. This is widely -accepted as good behavior. - -Instead of C and C, you could use C<_utf8_on> and C<_utf8_off>. -But this is, contrary to C<:utf8>, considered bad style. - -There are some shortcuts for oneliners; see C<-C> in L. - -=head3 What if I don't know which encoding was used? - -Do whatever you can to find out, and if you have to: guess. (Don't forget to -document your guess with a comment.) - -You could open the document in a web browser, and change the character set or -character encoding until you can visually confirm that all characters look the -way they should. - -There is no way to reliably detect the encoding automatically, so if people -keep sending you data without charset indication, you may have to educate them. - -=head3 Can I use Unicode in my Perl sources? - -Yes, you can! If your sources are UTF-8 encoded, you can indicate that with the -C pragma. - - use utf8; - -This doesn't do anything to your input, or to your output. It only influences -the way your sources are read. You can use Unicode in string literals, in -identifiers (but they still have to be "word characters" according to C<\w>), -and even in custom delimiters. - -=head3 Data::Dumper doesn't restore the UTF-8 flag; is it broken? - -No, Data::Dumper's Unicode abilities are as they should be. There have been -some complaints that it should restore the UTF-8 flag when the data is read -again with C. However, you should really not look at the flag, and -nothing indicates that Data::Dumper should break this rule. - -Here's what happens: when Perl reads in a string literal, it sticks to 8 bit -encoding as long as it can. (But perhaps originally it was internally encoded -as UTF-8, when you dumped it.) When it has to give that up because other -characters are added to the text string, it silently upgrades the string to -UTF-8. - -If you properly encode your strings for output, none of this is of your -concern, and you can just C dumped data as always. - -=head3 How can I determine if a string is a text string or a binary string? - -You can't. Some use the UTF-8 flag for this, but that's misuse, and makes well -behaved modules like Data::Dumper look bad. The flag is useless for this -purpose, because it's off when an 8 bit encoding (by default ISO-8859-1) is -used to store the string. - -This is something you, the programmer, has to keep track of; sorry. You could -consider adopting a kind of "Hungarian notation" to help with this. - -=head3 How do I convert from encoding FOO to encoding BAR? - -By first converting the FOO-encoded byte string to a text string, and then the -text string to a BAR-encoded byte string: - - my $text_string = decode('FOO', $foo_string); - my $bar_string = encode('BAR', $text_string); - -or by skipping the text string part, and going directly from one binary -encoding to the other: - - use Encode qw(from_to); - from_to($string, 'FOO', 'BAR'); # changes contents of $string - -or by letting automatic decoding and encoding do all the work: - - open my $foofh, '<:encoding(FOO)', 'example.foo.txt'; - open my $barfh, '>:encoding(BAR)', 'example.bar.txt'; - print { $barfh } $_ while <$foofh>; - -=head3 What about the C pragma? - -Don't use it. It makes no sense to deal with bytes in a text string, and it -makes no sense to deal with characters in a byte string. Do the proper -conversions (by decoding/encoding), and things will work out well: you get -character counts for decoded data, and byte counts for encoded data. - -C is usually a failed attempt to do something useful. Just forget -about it. - -=head3 What are C and C? - -These are alternate syntaxes for C and C. - -=head3 What's the difference between C and C? - -C is the official standard. C is Perl's way of being liberal in -what it accepts. If you have to communicate with things that aren't so liberal, -you may want to consider using C. If you have to communicate with things -that are too liberal, you may have to use C. The full explanation is in -L. - -C is internally known as C. This tutorial uses UTF-8 -consistently, even where utf8 is actually used internally, because the -distinction can be hard to make, and is mostly irrelevant. - -Okay, if you insist: the "internal format" is utf8, not UTF-8. (When it's not -some other encoding.) - -=head3 I lost track; what encoding is the internal format really? - -It's good that you lost track, because you shouldn't depend on the internal -format being any specific encoding. But since you asked: by default, the -internal format is either ISO-8859-1 (latin-1), or utf8, depending on the -history of the string. - -Perl knows how it stored the string internally, and will use that knowledge -when you C. In other words: don't try to find out what the internal -encoding for a certain string is, but instead just encode it into the encoding -that you want. - -=head3 What character encodings does Perl support? - -To find out which character encodings your Perl supports, run: - - perl -MEncode -le "print for Encode->encodings(':all')" - -=head3 Which version of perl should I use? - -Well, if you can, upgrade to the most recent, but certainly C<5.8.1> or newer. -This tutorial is based on the status quo as of C<5.8.7>. - -You should also check your modules, and upgrade them if necessary. For example, -HTML::Entities requires version >= 1.32 to function correctly, even though the -changelog is silent about this. - =head1 SUMMARY Decode everything you receive, encode everything you send out. (If it's text data.) +=head1 Q and A (or FAQ) + +After reading this document, you ought to read L too. + =head1 ACKNOWLEDGEMENTS Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the @@ -417,9 +200,9 @@ Gray. =head1 AUTHOR -Juerd Waalboer +Juerd Waalboer <#####@juerd.nl> =head1 SEE ALSO -L, L, L +L, L, L, L