perl's encoding on output by use of the ":encoding(...)" layer.
See L<open>.
-To mark the Perl source itself as being in an particular encoding,
+To mark the Perl source itself as being in a particular encoding,
see L<encoding>.
=item Regular Expressions
BidiWS Whitespace
BidiON Other Neutrals
+=back
+
=head2 Scripts
The scripts available for C<\p{In...}> and C<\P{In...}>, for example
Yi Radicals
Yi Syllables
+=over 4
+
=item *
The special pattern C<\X> match matches any extended Unicode sequence
=back
-What doesn't yet work are the followng cases:
+What doesn't yet work are the following cases:
=over 8
2.2 Categories - done [3][4]
2.3 Subtraction - MISSING [5][6]
2.4 Simple Word Boundaries - done [7]
- 2.5 Simple Loose Matches - MISSING [8]
+ 2.5 Simple Loose Matches - done [8]
2.6 End of Line - MISSING [9][10]
[ 1] \x{...}
[ 2] \N{...}
[ 3] . \p{Is...} \P{Is...}
- [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
+ [ 4] now scripts (see UTR#24 Script Names) in addition to blocks
[ 5] have negation
- [ 6] can use look-ahead to emulate subtracion
+ [ 6] can use look-ahead to emulate subtraction (*)
[ 7] include Letters in word characters
- [ 8] see UTR#21 Case Mappings
+ [ 8] see UTR#21 Case Mappings: Perl implements 1:1 mappings
[ 9] see UTR#13 Unicode Newline Guidelines
[10] should do ^ and $ also on \x{2028} and \x{2029}
+(*) Instead of [\u0370-\u03FF-[{UNASSIGNED}]] as suggested by the TR
+18 you can use negated lookahead: to match currently assigned modern
+Greek characters use for example
+
+ /(?!\p{Cn})[\x{0370}-\x{03ff}]/
+
+In other words: the matched character must not be a non-assigned
+character, but it must be in the block of modern Greek characters.
+
=item *
Level 2 - Extended Unicode Support
=head2 Unicode Encodings
Unicode characters are assigned to I<code points> which are abstract
-numbers. To use this numbers various encodings are needed.
+numbers. To use these numbers various encodings are needed.
=over 4
=item UTF-8
-UTF-8 is the encoding used internally by Perl. UTF-8 is variable
+UTF-8 is the encoding used internally by Perl. UTF-8 is a variable
length (1 to 6 bytes, current character allocations require 4 bytes),
-byteorder independent encoding. For ASCII UTF-8 is transparent
-(and we really mean 7-bit ASCII, not any 8-bit encoding).
+byteorder independent encoding. For ASCII, UTF-8 is transparent
+(and we really do mean 7-bit ASCII, not any 8-bit encoding).
+
+The following table is from Unicode 3.1.
+
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+
+ U+0000..U+007F 00..7F
+ U+0080..U+07FF C2..DF 80..BF
+ U+0800..U+0FFF E0 A0..BF 80..BF
+ U+1000..U+FFFF E1..EF 80..BF 80..BF
+ U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
+ U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
+ U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
+
+Or, another way to look at it, as bits:
+
+ Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
+
+ 0aaaaaaa 0aaaaaaa
+ 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
+ ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
+ 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
+
+As you can see, the continuation bytes all begin with C<10>, and the
+leading bits of the start byte tells how many bytes the are in the
+encoded character.
=item UTF-16, UTF-16BE, UTF16-LE, Surrogates, and BOMs (Byte Order Marks)
$uni = 0x10000 + ($hi - 0xD8000) * 0x400 + ($lo - 0xDC00);
-Because of the 16-bitness, UTF-16 is byteorder dependent. The UTF-16
+Because of the 16-bitness, UTF-16 is byteorder dependent. UTF-16
itself can be used for in-memory computations, but if storage or
-transfer is required, either the UTF-16BE (Big Endian), or UTF-16LE
+transfer is required, either UTF-16BE (Big Endian) or UTF-16LE
(Little Endian) must be chosen.
This introduces another problem: what if you just know that your data
is UTF-16, but you don't know which endianness? Byte Order Marks
(BOMs) are a solution to this. A special character has been reserved
-in Unicode to function as a byte order marker: the 0xFFFE is the BOM.
+in Unicode to function as a byte order marker: the character with the
+code point 0xFEFF is the BOM.
+
The trick is that if you read a BOM, you will know the byte order,
since if it was written on a big endian platform, you will read the
-bytes 0xFF 0xFE, but if it was written on a little endian platform,
-you will read the bytes 0xFE 0xFF. (And if the originating platform
-was writing in UTF-8, you will read the bytes 0xEF 0xBF 0xBE.)
+bytes 0xFE 0xFF, but if it was written on a little endian platform,
+you will read the bytes 0xFF 0xFE. (And if the originating platform
+was writing in UTF-8, you will read the bytes 0xEF 0xBB 0xBF.)
+
+The way this trick works is that the character with the code point
+0xFFFE is guaranteed not to be a valid Unicode character, so the
+sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in
+little-endian format" and cannot be "0xFFFE, represented in big-endian
+format".
=item UTF-32, UTF-32BE, UTF32-LE
The UTF-32 family is pretty much like the UTF-16 family, expect that
-the units are 32-bit, and therefore the surrogate scheme is not needed.
+the units are 32-bit, and therefore the surrogate scheme is not
+needed. The BOM signatures will be 0x00 0x00 0xFE 0xFF for BE and
+0xFF 0xFE 0x00 0x00 for LE.
=item UCS-2, UCS-4
-Encodings defined by the ISO 10646 standard. UCS-2 is 16-bit
-encoding, UCS-4 is 32-bit encoding. Unlike the UTF-16 the UCS-2
-is not extensible beyond 0xFFFF.
+Encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit
+encoding, UCS-4 is a 32-bit encoding. Unlike UTF-16, UCS-2
+is not extensible beyond 0xFFFF, because it does not use surrogates.
=item UTF-7
A seven-bit safe (non-eight-bit) encoding, useful if the
transport/storage is not eight-bit safe. Defined by RFC 2152.
+=back
+
+=head2 Security Implications of Malformed UTF-8
+
+Unfortunately, the specification of UTF-8 leaves some room for
+interpretation of how many bytes of encoded output one should generate
+from one input Unicode character. Strictly speaking, one is supposed
+to always generate the shortest possible sequence of UTF-8 bytes,
+because otherwise there is potential for input buffer overflow at the
+receiving end of a UTF-8 connection. Perl always generates the shortest
+length UTF-8, and with warnings on (C<-w> or C<use warnings;>) Perl will
+warn about non-shortest length UTF-8 (and other malformations, too,
+such as the surrogates, which are not real character code points.)
+
=head2 Unicode in Perl on EBCDIC
The way Unicode is handled on EBCDIC platforms is still rather
-experimental. On such a platform references to UTF-8 encoding in this
+experimental. On such a platform, references to UTF-8 encoding in this
document and elsewhere should be read as meaning UTF-EBCDIC as
specified in Unicode Technical Report 16 unless ASCII vs EBCDIC issues
are specifically discussed. There is no C<utfebcdic> pragma or
-":utfebcdic" layer, rather "utf8" and ":utf8" are re-used to mean
-platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic> for
-more discussion of the issues.
+":utfebcdic" layer, rather, "utf8" and ":utf8" are re-used to mean
+the platform's "natural" 8-bit encoding of Unicode. See L<perlebcdic>
+for more discussion of the issues.
+
+=head2 Using Unicode in XS
+
+If you want to handle Perl Unicode in XS extensions, you may find
+the following C APIs useful:
+
+=over 4
+
+=item *
+
+DO_UTF8(sv) returns true if the UTF8 flag is on and the bytes
+pragma is not in effect. SvUTF8(sv) returns true is the UTF8
+flag is on, the bytes pragma is ignored. Remember that UTF8
+flag being on does not mean that there would be any characters
+of code points greater than 255 or 127 in the scalar, or that
+there even are any characters in the scalar. The UTF8 flag
+means that any characters added to the string will be encoded
+in UTF8 if the code points of the characters are greater than
+255. Not "if greater than 127", since Perl's Unicode model
+is not to use UTF-8 until it's really necessary.
+
+=item *
+
+uvuni_to_utf8(buf, chr) writes a Unicode character code point into a
+buffer encoding the code poinqt as UTF-8, and returns a pointer
+pointing after the UTF-8 bytes.
+
+=item *
+
+utf8_to_uvuni(buf, lenp) reads UTF-8 encoded bytes from a buffer and
+returns the Unicode character code point (and optionally the length of
+the UTF-8 byte sequence).
+
+=item *
+
+utf8_length(s, len) returns the length of the UTF-8 encoded buffer in
+characters. sv_len_utf8(sv) returns the length of the UTF-8 encoded
+scalar.
+
+=item *
+
+sv_utf8_upgrade(sv) converts the string of the scalar to its UTF-8
+encoded form. sv_utf8_downgrade(sv) does the opposite (if possible).
+sv_utf8_encode(sv) is like sv_utf8_upgrade but the UTF8 flag does not
+get turned on. sv_utf8_decode() does the opposite of sv_utf8_encode().
+
+=item *
+
+is_utf8_char(buf) returns true if the buffer points to valid UTF-8.
+
+=item *
+
+is_utf8_string(buf, len) returns true if the len bytes of the buffer
+are valid UTF-8.
+
+=item *
+
+UTF8SKIP(buf) will return the number of bytes in the UTF-8 encoded
+character in the buffer. UNISKIP(chr) will return the number of bytes
+required to UTF-8-encode the Unicode character code point.
+
+=item *
+
+utf8_distance(a, b) will tell the distance in characters between the
+two pointers pointing to the same UTF-8 encoded buffer.
+
+=item *
+
+utf8_hop(s, off) will return a pointer to an UTF-8 encoded buffer that
+is C<off> (positive or negative) Unicode characters displaced from the
+UTF-8 buffer C<s>.
=back
+For more information, see L<perlapi>, and F<utf8.c> and F<utf8.h>
+in the Perl source code distribution.
+
=head1 SEE ALSO
-L<encoding>, L<Encode>, L<open>, L<bytes>, L<utf8>, L<perlretut>,
-L<perlvar/"${^WIDE_SYSTEM_CALLS}">
+L<perluniintro>, L<encoding>, L<Encode>, L<open>, L<utf8>, L<bytes>,
+L<perlretut>, L<perlvar/"${^WIDE_SYSTEM_CALLS}">
=cut
projects / p5sagit/p5-mst-13.2.git / blobdiff