Naturally, C<ord()> will do the reverse: turn a character to a code point.
-Note that C<\x..>, C<\x{..}> and C<chr(...)> for arguments less than
-0x100 (decimal 256) will generate an eight-bit character for backward
-compatibility with older Perls. For arguments of 0x100 or more,
-Unicode will always be produced. If you want UTF-8 always, use
-C<pack("U", ...)> instead of C<\x..>, C<\x{..}>, or C<chr()>.
+Note that C<\x..> (no C<{}> and only two hexadecimal digits), C<\x{...}>
+and C<chr(...)> for arguments less than 0x100 (decimal 256) will
+generate an eight-bit character for backward compatibility with older
+Perls. For arguments of 0x100 or more, Unicode will always be
+produced. If you want UTF-8 always, use C<pack("U", ...)> instead of
+C<\x..>, C<\x{...}>, or C<chr()>.
You can also use the C<charnames> pragma to invoke characters
by name in doublequoted strings:
my $georgian_an = pack("U", 0x10a0);
+Note that both C<\x{...}> and C<\N{...}> are compile-time string
+constants: you cannot use variables in them. if you want similar
+run-time functionality, use C<chr()> and C<charnames::vianame()>.
+
=head2 Handling Unicode
Handling Unicode is for the most part transparent: just use the
Normally writing out Unicode data
- print chr(0x100), "\n";
+ print FH chr(0x100), "\n";
+
+will print out the raw UTF-8 bytes, but you will get a warning
+out of that if you use C<-w> or C<use warnings>. To avoid the
+warning open the stream explicitly in UTF-8:
+
+ open FH, ">:utf8", "file";
-will print out the raw UTF-8 bytes.
+and on already open streams use C<binmode()>:
-But reading in correctly formed UTF-8 data will not magically turn
+ binmode(STDOUT, ":utf8");
+
+Reading in correctly formed UTF-8 data will not magically turn
the data into Unicode in Perl's eyes.
You can use either the C<':utf8'> I/O discipline when opening files
The I/O disciplines can also be specified more flexibly with
the C<open> pragma; see L<open>:
- use open ':utf8'; # input and output will be UTF-8
- open X, ">utf8";
- print X chr(0x100), "\n"; # this would have been UTF-8 without the pragma
+ use open ':utf8'; # input and output default discipline will be UTF-8
+ open X, ">file";
+ print X chr(0x100), "\n";
close X;
- open Y, "<utf8";
+ open Y, "<file";
printf "%#x\n", ord(<Y>); # this should print 0x100
close Y;
With the C<open> pragma you can use the C<:locale> discipline
- $ENV{LANG} = 'ru_RU.KOI8-R';
- # the :locale will probe the locale environment variables like LANG
+ $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R';
+ # the :locale will probe the locale environment variables like LC_ALL
use open OUT => ':locale'; # russki parusski
open(O, ">koi8");
print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
You can switch encodings on an already opened stream by using
C<binmode()>, see L<perlfunc/binmode>.
-The C<:locale> does not currently work with C<open()> and
-C<binmode()>, only with the C<open> pragma. The C<:utf8> and
-C<:encoding(...)> do work with all of C<open()>, C<binmode()>,
-and the C<open> pragma.
+The C<:locale> does not currently (as of Perl 5.8.0) work with
+C<open()> and C<binmode()>, only with the C<open> pragma. The
+C<:utf8> and C<:encoding(...)> do work with all of C<open()>,
+C<binmode()>, and the C<open> pragma.
Similarly, you may use these I/O disciplines on input streams to
automatically convert data from the specified encoding when it is
close F;
If you run this code twice, the contents of the F<file> will be twice
-UTF-8 encoded. A C<use open ':utf8'> would have avoided the bug.
+UTF-8 encoded. A C<use open ':utf8'> would have avoided the bug, or
+explicitly opening also the F<file> for input as UTF-8.
+
+B<NOTE>: the C<:utf8> and C<:encoding> features work only if your
+Perl has been built with the new "perlio" feature. Almost all
+Perl 5.8 platforms do use "perlio", though: you can see whether
+yours is by running "perl -V" and looking for C<useperlio=define>.
+
+=head2 Displaying Unicode As Text
+
+Sometimes you might want to display Perl scalars containing Unicode as
+simple ASCII (or EBCDIC) text. The following subroutine will convert
+its argument so that Unicode characters with code points greater than
+255 are displayed as "\x{...}", control characters (like "\n") are
+displayed as "\x..", and the rest of the characters as themselves.
+
+sub nice_string {
+ join("",
+ map { $_ > 255 ? # if wide character...
+ sprintf("\\x{%x}", $_) : # \x{...}
+ chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
+ sprintf("\\x%02x", $_) : # \x..
+ chr($_) } # else as themselves
+ unpack("U*", $_[0])); # unpack Unicode characters
+}
+
+For example, C<nice_string("foo\x{100}bar\n")> will return
+C<"foo\x{100}bar\x0a">.
=head2 Special Cases
The question of string equivalence turns somewhat complicated
in Unicode: what do you mean by equal?
- Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
- C<LATIN CAPITAL LETTER A>?
+(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to
+C<LATIN CAPITAL LETTER A>?)
The short answer is that by default Perl compares equivalence
(C<eq>, C<ne>) based only on code points of the characters.
People like to see their strings nicely sorted, or as Unicode
parlance goes, collated. But again, what do you mean by collate?
- Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
- C<LATIN CAPITAL LETTER A WITH GRAVE>?
+(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after
+C<LATIN CAPITAL LETTER A WITH GRAVE>?)
The short answer is that by default Perl compares strings (C<lt>,
C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the
=back
+=head1 UNICODE IN OLDER PERLS
+
+If you cannot upgrade your Perl to 5.8.0 or later, you can still
+do some Unicode processing by using the modules C<Unicode::String>,
+C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN.
+If you have the GNU recode installed, you can also use the
+Perl frontend C<Convert::Recode> for character conversions.
+
=head1 SEE ALSO
L<perlunicode>, L<Encode>, L<encoding>, L<open>, L<utf8>, L<bytes>,
projects / p5sagit/p5-mst-13.2.git / blobdiff