Message-ID: <f1gj4usu5m76bv88a3ldptnmo6ld7d44ri@4ax.com>

[p5sagit/p5-mst-13.2.git] / lib / encoding.pm

diff --git a/lib/encoding.pm b/lib/encoding.pm
index fd9d19b..642726d 100644 (file)
--- a/lib/encoding.pm
+++ b/lib/encoding.pm
@@ -1,5 +1,7 @@
 package encoding;
 
+our $VERSION = '1.00';
+
 use Encode;
 
 sub import {
@@ -43,7 +45,7 @@ encoding - pragma to control the conversion of legacy data into Unicode
 
     print "tera\n" if ord(pack("C", 0xdf)) == 0x3af;
 
-    # but pack/unpack C are not, in case you still
+    # but pack/unpack are not affected, in case you still
     # want back to your native encoding
 
     print "peta\n" if unpack("C", (pack("C", 0xdf))) == 0xdf;
@@ -57,17 +59,40 @@ encoding pragma you can change this default.
 The pragma is a per script, not a per block lexical.  Only the last
 C<use encoding> matters, and it affects B<the whole script>.
 
+Notice that only literals (string or regular expression) having only
+legacy code points are affected: if you mix data like this
+
+       \xDF\x{100}
+
+the data is assumed to be in (Latin 1 and) Unicode, not in your native
+encoding.  In other words, this will match in "greek":
+
+       "\xDF" =~ /\x{3af}/
+
+but this will not
+
+       "\xDF\x{100}" =~ /\x{3af}\x{100}/
+
+since the C<\xDF> on the left will B<not> be upgraded to C<\x{3af}>
+because of the C<\x{100}> on the left.  You should not be mixing your
+legacy data and Unicode in the same string.
+
+This pragma also affects encoding of the 0x80..0xFF code point range:
+normally characters in that range are left as eight-bit bytes (unless
+they are combined with characters with code points 0x100 or larger,
+in which case all characters need to become UTF-8 encoded), but if
+the C<encoding> pragma is present, even the 0x80..0xFF range always
+gets UTF-8 encoded.
+
 If no encoding is specified, the environment variable L<PERL_ENCODING>
-is consulted.  If that fails, "latin1" (ISO 8859-1) is assumed.
-If no encoding can be found, C<Unknown encoding '...'> error will be thrown.
+is consulted.  If that fails, "latin1" (ISO 8859-1) is assumed.  If no
+encoding can be found, C<Unknown encoding '...'> error will be thrown.
 
 =head1 KNOWN PROBLEMS
 
-The C<\x..> and C<\0...> in regular expressions are not affected by
-this pragma.  They very probably should.
-
-The charnames pragma ("\N{LATIN SMALL SHARP LETTER S}") does not work
-with this pragma.
+For native multibyte encodings (either fixed or variable length)
+the current implementation of the regular expressions may introduce
+recoding errors for longer regular expression literals than 127 bytes.
 
 =head1 SEE ALSO