=item *
-The special pattern C<\X> match matches any extended Unicode sequence
+The special pattern C<\X> matches any extended Unicode sequence
(a "combining character sequence" in Standardese), where the first
character is a base character and subsequent characters are mark
characters that apply to the base character. It is equivalent to
See L<Encode>.
-=head1 CAVEATS
-
-Whether an arbitrary piece of data will be treated as "characters" or
-"bytes" by internal operations cannot be divined at the current time.
-
-Use of locales with Unicode data may lead to odd results. Currently
-there is some attempt to apply 8-bit locale info to characters in the
-range 0..255, but this is demonstrably incorrect for locales that use
-characters above that range when mapped into Unicode. It will also
-tend to run slower. Avoidance of locales is strongly encouraged.
-
-=head1 UNICODE REGULAR EXPRESSION SUPPORT LEVEL
+=head2 Unicode Regular Expression Support Level
The following list of Unicode regular expression support describes
feature by feature the Unicode support implemented in Perl as of Perl
=over 4
-=item UTF-8
+=item *
+
+UTF-8
UTF-8 is a variable-length (1 to 6 bytes, current character allocations
require 4 bytes), byteorder independent encoding. For ASCII, UTF-8 is
transparent (and we really do mean 7-bit ASCII, not another 8-bit encoding).
-The following table is from Unicode 3.1.
+The following table is from Unicode 3.2.
Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
- U+0000..U+007F 00..7F
- U+0080..U+07FF C2..DF 80..BF
+ U+0000..U+007F 00..7F
+ U+0080..U+07FF C2..DF 80..BF
U+0800..U+0FFF E0 A0..BF 80..BF
- U+1000..U+FFFF E1..EF 80..BF 80..BF
+ U+1000..U+CFFF E1..EC 80..BF 80..BF
+ U+D000..U+D7FF ED 80..9F 80..BF
+ U+D800..U+DFFF ******* ill-formed *******
+ U+E000..U+FFFF EE..EF 80..BF 80..BF
U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
+Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
+the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
Or, another way to look at it, as bits:
Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
As you can see, the continuation bytes all begin with C<10>, and the
-leading bits of the start byte tells how many bytes the are in the
+leading bits of the start byte tell how many bytes the are in the
encoded character.
-=item UTF-EBCDIC
+=item *
+
+UTF-EBCDIC
Like UTF-8, but EBCDIC-safe, as UTF-8 is ASCII-safe.
-=item UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
+=item *
+
+UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
(The followings items are mostly for reference, Perl doesn't
use them internally.)
little-endian format" and cannot be "0xFFFE, represented in big-endian
format".
-=item UTF-32, UTF-32BE, UTF32-LE
+=item *
+
+UTF-32, UTF-32BE, UTF32-LE
The UTF-32 family is pretty much like the UTF-16 family, expect that
the units are 32-bit, and therefore the surrogate scheme is not
needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
0xFF 0xFE 0x00 0x00 for LE.
-=item UCS-2, UCS-4
+=item *
+
+UCS-2, UCS-4
Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
is not extensible beyond 0xFFFF, because it does not use surrogates.
-=item UTF-7
+=item *
+
+UTF-7
A seven-bit safe (non-eight-bit) encoding, useful if the
transport/storage is not eight-bit safe. Defined by RFC 2152.
=head2 Using Unicode in XS
If you want to handle Perl Unicode in XS extensions, you may find
-the following C APIs useful:
+the following C APIs useful (see perlapi for details):
=over 4
=item *
-DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes
-pragma is not in effect. SvUTF8(sv) returns true is the UTF8
-flag is on, the bytes pragma is ignored. Remember that UTF8
-flag being on does not mean that there would be any characters
-of code points greater than 255 or 127 in the scalar, or that
-there even are any characters in the scalar. The UTF8 flag
-means that any characters added to the string will be encoded
-in UTF8 if the code points of the characters are greater than
-255. Not "if greater than 127", since Perl's Unicode model
-is not to use UTF-8 until it's really necessary.
+DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes pragma
+is not in effect. SvUTF8(sv) returns true is the UTF8 flag is on, the
+bytes pragma is ignored. The UTF8 flag being on does B<not> mean that
+there are any characters of code points greater than 255 (or 127) in
+the scalar, or that there even are any characters in the scalar.
+What the UTF8 flag means is that the sequence of octets in the
+representation of the scalar is the sequence of UTF-8 encoded
+code points of the characters of a string. The UTF8 flag being
+off means that each octet in this representation encodes a single
+character with codepoint 0..255 within the string. Perl's Unicode
+model is not to use UTF-8 until it's really necessary.
=item *
=item *
-utf8_length(s, len) returns the length of the UTF-8 encoded buffer in
-characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
+utf8_length(start, end) returns the length of the UTF-8 encoded buffer
+in characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
scalar.
=item *
encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
+Note that none of these are to be used as general purpose encoding/decoding
+interfaces: use Encode for that. sv_utf8_upgrade() is affected by the
+encoding pragma, but sv_utf8_downgrade() is not (since the encoding
+pragma is designed to be a one-way street).
=item *
-is_utf8_char(buf) returns true if the buffer points to valid UTF-8.
+is_utf8_char(s) returns true if the pointer points to a valid UTF-8
+character.
=item *
UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
character in the buffer. UNISKIP(chr) will return the number of bytes
-required to UTF-8-encode the Unicode character code point.
+required to UTF-8-encode the Unicode character code point. UTF8SKIP()
+is useful for example for iterating over the characters of a UTF-8
+encoded buffer; UNISKIP() is useful for example in computing
+the size required for a UTF-8 encoded buffer.
=item *
utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
is C<off> (positive or negative) Unicode characters displaced from the
-UTF-8 buffer C<s>.
+UTF-8 buffer C<s>. Be careful not to overstep the buffer: utf8_hop()
+will merrily run off the end or the beginning if told to do so.
=item *
pv_uni_display(dsv, spv, len, pvlim, flags) and sv_uni_display(dsv,
ssv, pvlim, flags) are useful for debug output of Unicode strings and
-scalars (only for debug: they display B<all> characters as hexadecimal
-code points).
+scalars. By default they are useful only for debug: they display
+B<all> characters as hexadecimal code points, but with the flags
+UNI_DISPLAY_ISPRINT and UNI_DISPLAY_BACKSLASH you can make the output
+more readable.
=item *
-ibcmp_utf8(s1, u1, len1, s2, u2, len2) can be used to compare two
-strings case-insensitively in Unicode. (For case-sensitive
-comparisons you can just use memEQ() and memNE() as usual.)
+ibcmp_utf8(s1, pe1, u1, l1, u1, s2, pe2, l2, u2) can be used to
+compare two strings case-insensitively in Unicode.
+(For case-sensitive comparisons you can just use memEQ() and memNE()
+as usual.)
=back
For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
in the Perl source code distribution.
+=head1 BUGS
+
+Use of locales with Unicode data may lead to odd results. Currently
+there is some attempt to apply 8-bit locale info to characters in the
+range 0..255, but this is demonstrably incorrect for locales that use
+characters above that range when mapped into Unicode. It will also
+tend to run slower. Avoidance of locales is strongly encouraged.
+
+Some functions are slower when working on UTF-8 encoded strings than
+on byte encoded strings. All functions that need to hop over
+characters such as length(), substr() or index() can work B<much>
+faster when the underlying data are byte-encoded. Witness the
+following benchmark:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-2,{
+ LENGTH_B => q{ length($b) },
+ LENGTH_U => q{ length($u) },
+ SUBSTR_B => q{ substr($b, $l/4, $l/2) },
+ SUBSTR_U => q{ substr($u, $l/4, $l/2) },
+ });
+ '
+ Benchmark: running LENGTH_B, LENGTH_U, SUBSTR_B, SUBSTR_U for at least 2 CPU seconds...
+ LENGTH_B: 2 wallclock secs ( 2.36 usr + 0.00 sys = 2.36 CPU) @ 5649983.05/s (n=13333960)
+ LENGTH_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 12155.45/s (n=25648)
+ SUBSTR_B: 3 wallclock secs ( 2.16 usr + 0.00 sys = 2.16 CPU) @ 374480.09/s (n=808877)
+ SUBSTR_U: 2 wallclock secs ( 2.11 usr + 0.00 sys = 2.11 CPU) @ 6791.00/s (n=14329)
+
+The numbers show an incredible slowness on long UTF-8 strings and you
+should carefully avoid to use these functions within tight loops. For
+example if you want to iterate over characters, it is infinitely
+better to split into an array than to use substr, as the following
+benchmark shows:
+
+ % perl -e '
+ use Benchmark;
+ use strict;
+ our $l = 10000;
+ our $u = our $b = "x" x $l;
+ substr($u,0,1) = "\x{100}";
+ timethese(-5,{
+ SPLIT_B => q{ for my $c (split //, $b){} },
+ SPLIT_U => q{ for my $c (split //, $u){} },
+ SUBSTR_B => q{ for my $i (0..length($b)-1){my $c = substr($b,$i,1);} },
+ SUBSTR_U => q{ for my $i (0..length($u)-1){my $c = substr($u,$i,1);} },
+ });
+ '
+ Benchmark: running SPLIT_B, SPLIT_U, SUBSTR_B, SUBSTR_U for at least 5 CPU seconds...
+ SPLIT_B: 6 wallclock secs ( 5.29 usr + 0.00 sys = 5.29 CPU) @ 56.14/s (n=297)
+ SPLIT_U: 5 wallclock secs ( 5.17 usr + 0.01 sys = 5.18 CPU) @ 55.21/s (n=286)
+ SUBSTR_B: 5 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 123.22/s (n=658)
+ SUBSTR_U: 7 wallclock secs ( 6.20 usr + 0.00 sys = 6.20 CPU) @ 0.81/s (n=5)
+
+You see, the algorithm based on substr() was faster with byte encoded
+data but it is pathologically slow with UTF-8 data.
+
=head1 SEE ALSO
L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
projects / p5sagit/p5-mst-13.2.git / blobdiff