pod2html: try to be EOL agnostic.

[p5sagit/p5-mst-13.2.git] / lib / utf8.pm

diff --git a/lib/utf8.pm b/lib/utf8.pm
index cb19686..5bec955 100644 (file)
--- a/lib/utf8.pm
+++ b/lib/utf8.pm
@@ -35,7 +35,7 @@ utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
 
 The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
 program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
-platforms).  The C<no utf8> pragma tells Perl to switch back to treating 
+platforms).  The C<no utf8> pragma tells Perl to switch back to treating
 the source text as literal bytes in the current lexical scope.
 
 This pragma is primarily a compatibility device.  Perl versions
@@ -63,6 +63,12 @@ UTF-EBCDIC character.
 
 =back
 
+Note that if you have bytes with the eighth bit on in your script
+(for example embedded Latin-1 in your string literals), C<use utf8>
+will be unhappy since the bytes are most probably not well-formed
+UTF-8.  If you want to have such bytes and use utf8, you can disable
+utf8 until the end the block (or file, if at top level) by C<no utf8;>.
+
 =head2 Utility functions
 
 The following functions are defined in the C<utf8::> package by the perl core.
@@ -73,24 +79,47 @@ The following functions are defined in the C<utf8::> package by the perl core.
 
 Converts internal representation of string to the Perl's internal
 I<UTF-X> form.  Returns the number of octets necessary to represent
-the string as I<UTF-X>.
+the string as I<UTF-X>.  Note that this should not be used to convert
+a legacy byte encoding to Unicode: use Encode for that.  Affected
+by the encoding pragma.
 
 =item * utf8::downgrade($string[, CHECK])
 
 Converts internal representation of string to be un-encoded bytes.
+Note that this should not be used to convert Unicode back to a legacy
+byte encoding: use Encode for that.  B<Not> affected by the encoding
+pragma.
 
 =item * utf8::encode($string)
 
-Converts (in-place) I<$string> from logical characters to octet sequence
-representing it in Perl's I<UTF-X> encoding.
+Converts (in-place) I<$string> from logical characters to octet
+sequence representing it in Perl's I<UTF-X> encoding.  Note that this
+should not be used to convert a legacy byte encoding to Unicode: use
+Encode for that.
 
 =item * $flag = utf8::decode($string)
 
 Attempts to convert I<$string> in-place from Perl's I<UTF-X> encoding
-into logical characters.
+into logical characters.  Note that this should not be used to convert
+Unicode back to a legacy byte encoding: use Encode for that.
+
+=item * $flag = utf8::valid(STRING)
+
+[INTERNAL] Test whether STRING is in a consistent state.  Will return
+true if string is held as bytes, or is well-formed UTF-8 and has the
+UTF-8 flag on.  Main reason for this routine is to allow Perl's
+testsuite to check that operations have left strings in a consistent
+state.
 
 =back
 
+C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is cleared.
+See L<perlunicode> for more on the UTF8 flag and the C API functions
+C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
+and C<sv_utf8_decode>, which are wrapped by the Perl functions
+C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
+C<utf8::decode>.
+
 =head1 SEE ALSO
 
 L<perlunicode>, L<bytes>