3 perlunitut - Perl Unicode Tutorial
7 The days of just flinging strings around are over. It's well established that
8 modern programs need to be capable of communicating funny accented letters, and
9 things like euro symbols. This means that programmers need new habits. It's
10 easy to program Unicode capable software, but it does require discipline to do
13 There's a lot to know about character sets, and text encodings. It's probably
14 best to spend a full day learning all this, but the basics can be learned in
17 These are not the very basics, though. It is assumed that you already
18 know the difference between bytes and characters, and realise (and accept!)
19 that there are many different character sets and encodings, and that your
20 program has to be explicit about them. Recommended reading is "The Absolute
21 Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
22 and Character Sets (No Excuses!)" by Joel Spolsky, at
23 L<http://joelonsoftware.com/articles/Unicode.html>.
25 This tutorial speaks in rather absolute terms, and provides only a limited view
26 of the wealth of character string related features that Perl has to offer. For
27 most projects, this information will probably suffice.
31 It's important to set a few things straight first. This is the most important
32 part of this tutorial. This view may conflict with other information that you
33 may have found on the web, but that's mostly because many sources are wrong.
35 You may have to re-read this entire section a few times...
39 B<Unicode> is a character set with room for lots of characters. The ordinal
40 value of a character is called a B<code point>.
42 There are many, many code points, but computers work with bytes, and a byte can
43 have only 256 values. Unicode has many more characters, so you need a method
44 to make these accessible.
46 Unicode is encoded using several competing encodings, of which UTF-8 is the
47 most used. In a Unicode encoding, multiple subsequent bytes can be used to
48 store a single code point, or simply: character.
52 B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are
53 the same thing, but they're not. There are more Unicode encodings, but much of
54 the world has standardized on UTF-8.
56 UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
57 only one byte per character. All other characters are encoded as two or more
58 (up to six) bytes using a complex scheme. Fortunately, Perl handles this for
59 us, so we don't have to worry about this.
61 =head3 Text strings (character strings)
63 B<Text strings>, or B<character strings> are made of characters. Bytes are
64 irrelevant here, and so are encodings. Each character is just that: the
67 On a text string, you would do things like:
70 if ($string =~ /^\d+$/) { ... }
71 $text = ucfirst $text;
72 my $character_count = length $text;
74 The value of a character (C<ord>, C<chr>) is the corresponding Unicode code
77 =head3 Binary strings (byte strings)
79 B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have
80 characters, just bytes. All communication with the outside world (anything
81 outside of your current Perl process) is done in binary.
83 On a binary string, you would do things like:
85 my (@length_content) = unpack "(V/a)*", $binary;
86 $binary =~ s/\x00\x0F/\xFF\xF0/; # for the brave :)
88 my $byte_count = length $binary;
92 B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode,
93 you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>.
94 Some encodings, like the C<iso-8859> ("latin") range, do not support the full
95 Unicode standard; characters that can't be represented are lost in the
100 B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to
101 know what encoding was used during the encoding phase. And most of all, it must
102 be something decodable. It doesn't make much sense to decode a PNG image into a
105 =head3 Internal format
107 Perl has an B<internal format>, an encoding that it uses to encode text strings
108 so it can store them in memory. All text strings are in this internal format.
109 In fact, text strings are never in any other format!
111 You shouldn't worry about what this format is, because conversion is
112 automatically done when you decode or encode.
114 =head2 Your new toolkit
116 Add to your standard heading the following line:
118 use Encode qw(encode decode);
120 Or, if you're lazy, just:
124 =head2 I/O flow (the actual 5 minute tutorial)
126 The typical input/output flow of a program is:
128 1. Receive and decode
132 If your input is binary, and is supposed to remain binary, you shouldn't decode
133 it to a text string, of course. But in all other cases, you should decode it.
135 Decoding can't happen reliably if you don't know how the data was encoded. If
136 you get to choose, it's a good idea to standardize on UTF-8.
138 my $foo = decode('UTF-8', get 'http://example.com/');
139 my $bar = decode('ISO-8859-1', readline STDIN);
140 my $xyzzy = decode('Windows-1251', $cgi->param('foo'));
142 Processing happens as you knew before. The only difference is that you're now
143 using characters instead of bytes. That's very useful if you use things like
144 C<substr>, or C<length>.
146 It's important to realize that there are no bytes in a text string. Of course,
147 Perl has its internal encoding to store the string in memory, but ignore that.
148 If you have to do anything with the number of bytes, it's probably best to move
149 that part to step 3, just after you've encoded the string. Then you know
150 exactly how many bytes it will be in the destination string.
152 The syntax for encoding text strings to binary strings is as simple as decoding:
154 $body = encode('UTF-8', $body);
156 If you needed to know the length of the string in bytes, now's the perfect time
157 for that. Because C<$body> is now a byte string, C<length> will report the
158 number of bytes, instead of the number of characters. The number of
159 characters is no longer known, because characters only exist in text strings.
161 my $byte_count = length $body;
163 And if the protocol you're using supports a way of letting the recipient know
164 which character encoding you used, please help the receiving end by using that
165 feature! For example, E-mail and HTTP support MIME headers, so you can use the
166 C<Content-Type> header. They can also have C<Content-Length> to indicate the
167 number of I<bytes>, which is always a good idea to supply if the number is
170 "Content-Type: text/plain; charset=UTF-8",
171 "Content-Length: $byte_count"
175 =head3 This isn't really a Unicode tutorial, is it?
177 No, Perl has an abstracted interface for all supported character encodings, so
178 this is actually a generic C<Encode> tutorial. But many people think that
179 Unicode is special and magical, and I didn't want to disappoint them, so I
180 decided to call this document a Unicode tutorial.
182 =head3 What about binary data, like images?
184 Well, apart from a bare C<binmode $fh>, you shouldn't treat them specially.
185 (The binmode is needed because otherwise Perl may convert line endings on Win32
188 Be careful, though, to never combine text strings with binary strings. If you
189 need text in a binary stream, encode your text strings first using the
190 appropriate encoding, then join them with binary strings. See also: "What if I
193 =head3 What about the UTF-8 flag?
195 Please, unless you're hacking the internals, or debugging weirdness, don't
196 think about the UTF-8 flag at all. That means that you very probably shouldn't
197 use C<is_utf8>, C<_utf8_on> or C<_utf8_off> at all.
199 Perl's internal format happens to be UTF-8. Unfortunately, Perl can't keep a
200 secret, so everyone knows about this. That is the source of much confusion.
201 It's better to pretend that the internal format is some unknown encoding,
202 and that you always have to encode and decode explicitly.
204 =head3 When should I decode or encode?
206 Whenever you're communicating with anything that is external to your perl
207 process, like a database, a text file, a socket, or another program. Even if
208 the thing you're communicating with is also written in Perl.
210 =head3 What if I don't decode?
212 Whenever your encoded, binary string is used together with a text string, Perl
213 will assume that your binary string was encoded with ISO-8859-1, also known as
214 latin-1. If it wasn't latin-1, then your data is unpleasantly converted. For
215 example, if it was UTF-8, the individual bytes of multibyte characters are seen
216 as separate characters, and then again converted to UTF-8. Such double encoding
217 can be compared to double HTML encoding (C<&gt;>), or double URI encoding
220 This silent implicit decoding is known as "upgrading". That may sound
221 positive, but it's best to avoid it.
223 =head3 What if I don't encode?
225 Your text string will be sent using the bytes in Perl's internal format. In
226 some cases, Perl will warn you that you're doing something wrong, with a
229 Wide character in print at example.pl line 2.
231 Because the internal format is often UTF-8, these bugs are hard to spot,
232 because UTF-8 is usually the encoding you wanted! But don't be lazy, and don't
233 use the fact that Perl's internal format is UTF-8 to your advantage. Encode
234 explicitly to avoid weird bugs, and to show to maintenance programmers that you
235 thought this through.
237 =head3 Is there a way to automatically decode or encode?
239 If all data that comes from a certain handle is encoded in exactly the same
240 way, you can tell the PerlIO system to automatically decode everything, with
241 the C<encoding> layer. If you do this, you can't accidentally forget to decode
242 or encode anymore, on things that use the layered handle.
244 You can provide this layer when C<open>ing the file:
246 open my $fh, '>:encoding(UTF-8)', $filename; # auto encoding on write
247 open my $fh, '<:encoding(UTF-8)', $filename; # auto decoding on read
249 Or if you already have an open filehandle:
251 binmode $fh, ':encoding(UTF-8)';
253 Some database drivers for DBI can also automatically encode and decode, but
254 that is typically limited to the UTF-8 encoding, because they cheat.
256 =head3 Cheat?! Tell me, how can I cheat?
258 Well, because Perl's internal format is UTF-8, you can just skip the encoding
259 or decoding step, and manipulate the UTF-8 flag directly.
261 Instead of C<:encoding(UTF-8)>, you can simply use C<:utf8>. This is widely
262 accepted as good behavior.
264 Instead of C<decode> and C<encode>, you could use C<_utf8_on> and C<_utf8_off>.
265 But this is, contrary to C<:utf8>, considered bad style.
267 There are some shortcuts for oneliners; see C<-C> in L<perlrun>.
269 =head3 What if I don't know which encoding was used?
271 Do whatever you can to find out, and if you have to: guess. (Don't forget to
272 document your guess with a comment.)
274 You could open the document in a web browser, and change the character set or
275 character encoding until you can visually confirm that all characters look the
278 There is no way to reliably detect the encoding automatically, so if people
279 keep sending you data without charset indication, you may have to educate them.
281 =head3 Can I use Unicode in my Perl sources?
283 Yes, you can! If your sources are UTF-8 encoded, you can indicate that with the
288 This doesn't do anything to your input, or to your output. It only influences
289 the way your sources are read. You can use Unicode in string literals, in
290 identifiers (but they still have to be "word characters" according to C<\w>),
291 and even in custom delimiters.
293 =head3 Data::Dumper doesn't restore the UTF-8 flag; is it broken?
295 No, Data::Dumper's Unicode abilities are as they should be. There have been
296 some complaints that it should restore the UTF-8 flag when the data is read
297 again with C<eval>. However, you should really not look at the flag, and
298 nothing indicates that Data::Dumper should break this rule.
300 Here's what happens: when Perl reads in a string literal, it sticks to 8 bit
301 encoding as long as it can. (But perhaps originally it was internally encoded
302 as UTF-8, when you dumped it.) When it has to give that up because other
303 characters are added to the text string, it silently upgrades the string to
306 If you properly encode your strings for output, none of this is of your
307 concern, and you can just C<eval> dumped data as always.
309 =head3 How can I determine if a string is a text string or a binary string?
311 You can't. Some use the UTF-8 flag for this, but that's misuse, and makes well
312 behaved modules like Data::Dumper look bad. The flag is useless for this
313 purpose, because it's off when an 8 bit encoding (by default ISO-8859-1) is
314 used to store the string.
316 This is something you, the programmer, has to keep track of; sorry. You could
317 consider adopting a kind of "Hungarian notation" to help with this.
319 =head3 How do I convert from encoding FOO to encoding BAR?
321 By first converting the FOO-encoded byte string to a text string, and then the
322 text string to a BAR-encoded byte string:
324 my $text_string = decode('FOO', $foo_string);
325 my $bar_string = encode('BAR', $text_string);
327 or by skipping the text string part, and going directly from one binary
328 encoding to the other:
330 use Encode qw(from_to);
331 from_to($string, 'FOO', 'BAR'); # changes contents of $string
333 or by letting automatic decoding and encoding do all the work:
335 open my $foofh, '<:encoding(FOO)', 'example.foo.txt';
336 open my $barfh, '>:encoding(BAR)', 'example.bar.txt';
337 print { $barfh } $_ while <$foofh>;
339 =head3 What about the C<use bytes> pragma?
341 Don't use it. It makes no sense to deal with bytes in a text string, and it
342 makes no sense to deal with characters in a byte string. Do the proper
343 conversions (by decoding/encoding), and things will work out well: you get
344 character counts for decoded data, and byte counts for encoded data.
346 C<use bytes> is usually a failed attempt to do something useful. Just forget
349 =head3 What are C<decode_utf8> and C<encode_utf8>?
351 These are alternate syntaxes for C<decode('utf8', ...)> and C<encode('utf8',
354 =head3 What's the difference between C<UTF-8> and C<utf8>?
356 C<UTF-8> is the official standard. C<utf8> is Perl's way of being liberal in
357 what it accepts. If you have to communicate with things that aren't so liberal,
358 you may want to consider using C<UTF-8>. If you have to communicate with things
359 that are too liberal, you may have to use C<utf8>. The full explanation is in
362 C<UTF-8> is internally known as C<utf-8-strict>. This tutorial uses UTF-8
363 consistently, even where utf8 is actually used internally, because the
364 distinction can be hard to make, and is mostly irrelevant.
366 Okay, if you insist: the "internal format" is utf8, not UTF-8. (When it's not
367 some other encoding.)
369 =head3 I lost track; what encoding is the internal format really?
371 It's good that you lost track, because you shouldn't depend on the internal
372 format being any specific encoding. But since you asked: by default, the
373 internal format is either ISO-8859-1 (latin-1), or utf8, depending on the
374 history of the string.
376 Perl knows how it stored the string internally, and will use that knowledge
377 when you C<encode>. In other words: don't try to find out what the internal
378 encoding for a certain string is, but instead just encode it into the encoding
381 =head3 What character encodings does Perl support?
383 To find out which character encodings your Perl supports, run:
385 perl -MEncode -le "print for Encode->encodings(':all')"
387 =head3 Which version of perl should I use?
389 Well, if you can, upgrade to the most recent, but certainly C<5.8.1> or newer.
390 This tutorial is based on the status quo as of C<5.8.7>.
392 You should also check your modules, and upgrade them if necessary. For example,
393 HTML::Entities requires version >= 1.32 to function correctly, even though the
394 changelog is silent about this.
398 Decode everything you receive, encode everything you send out. (If it's text
401 =head1 ACKNOWLEDGEMENTS
403 Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the
404 Amsterdam Perl Mongers meetings got me interested and determined to find out
405 how to use character encodings in Perl in ways that don't break easily.
407 Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch
408 Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.
410 Thanks to the people who asked about this kind of stuff in several Perl IRC
411 channels, and have constantly reminded me that a simpler explanation was
414 Thanks to the people who reviewed this document for me, before it went public.
415 They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
420 Juerd Waalboer <juerd@cpan.org>
424 L<perlunicode>, L<perluniintro>, L<Encode>