Explain the "gaps" in the UTF-8 encoding.

p4raw-id: //depot/perl@15761

diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod
index ad3637f..26e704a 100644 (file)
--- a/pod/perlunicode.pod
+++ b/pod/perlunicode.pod
@@ -737,6 +737,11 @@ The following table is from Unicode 3.2.
 
 Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
 the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
+The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings:
+it is technically possible to UTF-8-encode a single code point in different
+ways, but that is explicitly forbidden, and the shortest possible encoding
+should always be used (and that is what Perl does).
+
 Or, another way to look at it, as bits:
 
  Code Points                    1st Byte   2nd Byte  3rd Byte  4th Byte

diff --git a/utf8.h b/utf8.h
index 3787832..0b74517 100644 (file)
--- a/utf8.h
+++ b/utf8.h
@@ -80,6 +80,10 @@ END_EXTERN_C
 
 Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF,
 the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF.
+The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings:
+it is technically possible to UTF-8-encode a single code point in different
+ways, but that is explicitly forbidden, and the shortest possible encoding
+should always be used (and that is what Perl does).
 
  */