A Unicode I<logical> "character" can actually consist of more than one internal
I<actual> "character" or code point. For Western languages, this is adequately
-represented by a I<base character> (like C<LATIN CAPITAL LETTER A>), followed
+modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed
by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of
base character and modifiers is called a I<combining character
sequence>. Some non-western languages require more complicated
-representations, so Unicode invented a I<grapheme cluster> and then an
-I<extended grapheme cluster>. For example, A Korean Hangul syllable is
+models, so Unicode created the I<grapheme cluster> concept, and then the
+I<extended grapheme cluster>. For example, a Korean Hangul syllable is
considered a single logical character, but most often consists of three actual
-characters: a leading consonant followed by an interior vowel followed by a
-trailing consonant.
+Unicode characters: a leading consonant followed by an interior vowel followed
+by a trailing consonant.
Whether to call these extended grapheme clusters "characters" depends on your
point of view. If you are a programmer, you probably would tend towards seeing
With this "whole sequence" view of characters, the total number of
characters is open-ended. But in the programmer's "one unit is one
character" point of view, the concept of "characters" is more
-deterministic. In this document, we take that second point of view:
+deterministic. In this document, we take that second point of view:
one "character" is one Unicode code point.
For some combinations, there are I<precomposed> characters.
A user of Perl does not normally need to know nor care how Perl
happens to encode its internal strings, but it becomes relevant when
-outputting Unicode strings to a stream without a PerlIO layer -- one with
-the "default" encoding. In such a case, the raw bytes used internally
+outputting Unicode strings to a stream without a PerlIO layer (one with
+the "default" encoding). In such a case, the raw bytes used internally
(the native character set or UTF-8, as appropriate for each string)
will be used, and a "Wide character" warning will be issued if those
strings contain a character beyond 0x00FF.
The I/O layers can also be specified more flexibly with
the C<open> pragma. See L<open>, or look at the following example.
- use open ':encoding(utf8)'; # input/output default encoding will be UTF-8
+ use open ':encoding(utf8)'; # input/output default encoding will be
+ # UTF-8
open X, ">file";
print X chr(0x100), "\n";
close X;
With the C<open> pragma you can use the C<:locale> layer
BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' }
- # the :locale will probe the locale environment variables like LC_ALL
+ # the :locale will probe the locale environment variables like
+ # LC_ALL
use open OUT => ':locale'; # russki parusski
open(O, ">koi8");
print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1
255 are displayed as C<\x{...}>, control characters (like C<\n>) are
displayed as C<\x..>, and the rest of the characters as themselves:
- sub nice_string {
- join("",
- map { $_ > 255 ? # if wide character...
- sprintf("\\x{%04X}", $_) : # \x{...}
- chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
- sprintf("\\x%02X", $_) : # \x..
- quotemeta(chr($_)) # else quoted or as themselves
+ sub nice_string {
+ join("",
+ map { $_ > 255 ? # if wide character...
+ sprintf("\\x{%04X}", $_) : # \x{...}
+ chr($_) =~ /[[:cntrl:]]/ ? # else if control character ...
+ sprintf("\\x%02X", $_) : # \x..
+ quotemeta(chr($_)) # else quoted or as themselves
} unpack("W*", $_[0])); # unpack Unicode characters
}
Character Ranges and Classes
-Character ranges in regular expression character classes (C</[a-z]/>)
-and in the C<tr///> (also known as C<y///>) operator are not magically
-Unicode-aware. What this means is that C<[A-Za-z]> will not magically start
-to mean "all alphabetic letters"; not that it does mean that even for
-8-bit characters, you should be using C</[[:alpha:]]/> in that case.
+Character ranges in regular expression bracketed character classes ( e.g.,
+C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not
+magically Unicode-aware. What this means is that C<[A-Za-z]> will not
+magically start to mean "all alphabetic letters" (not that it does mean that
+even for 8-bit characters; for those, if you are using locales (L<perllocale>),
+use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>).
-For specifying character classes like that in regular expressions,
-you can use the various Unicode properties--C<\pL>, or perhaps
-C<\p{Alphabetic}>, in this particular case. You can use Unicode
-code points as the end points of character ranges, but there is no
-magic associated with specifying a certain range. For further
-information--there are dozens of Unicode character classes--see
-L<perlunicode>.
+All the properties that begin with C<\p> (and its inverse C<\P>) are actually
+character classes that are Unicode-aware. There are dozens of them, see
+L<perluniprops>.
+
+You can use Unicode code points as the end points of character ranges, and the
+range will include all Unicode code points that lie between those end points.
=item *
How Do I Know Whether My String Is In Unicode?
You shouldn't have to care. But you may, because currently the semantics of the
-characters whose ordinals are in the range 128 to 255 is different depending on
+characters whose ordinals are in the range 128 to 255 are different depending on
whether the string they are contained within is in Unicode or not.
(See L<perlunicode/When Unicode Does Not Happen>.)
return the value of the internal "utf8ness" flag attached to the
C<$string>. If the flag is off, the bytes in the scalar are interpreted
as a single byte encoding. If the flag is on, the bytes in the scalar
-are interpreted as the (multi-byte, variable-length) UTF-8 encoded code
-points of the characters. Bytes added to an UTF-8 encoded string are
+are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded
+code points of the characters. Bytes added to a UTF-8 encoded string are
automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars
are merged (double-quoted interpolation, explicit concatenation, and
printf/sprintf parameter substitution), the result will be UTF-8 encoded
use bytes;
print length($unicode), "\n"; # will also print 2
# (the 0xC4 0x80 of the UTF-8)
+ no bytes;
=item *
You can find the bytes that make up a UTF-8 sequence with
- @bytes = unpack("C*", $Unicode_string)
+ @bytes = unpack("C*", $Unicode_string)
and you can create well-formed Unicode with
- $Unicode_string = pack("U*", 0xff, ...)
+ $Unicode_string = pack("U*", 0xff, ...)
=item *