From: Jarkko Hietaniemi Date: Sat, 6 Apr 2002 16:17:57 +0000 (+0000) Subject: Explain the "gaps" in the UTF-8 encoding. X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=commitdiff_plain;h=37361303e31c2073b3e9c4c5bce1a4cc42832f83;p=p5sagit%2Fp5-mst-13.2.git Explain the "gaps" in the UTF-8 encoding. p4raw-id: //depot/perl@15761 --- diff --git a/pod/perlunicode.pod b/pod/perlunicode.pod index ad3637f..26e704a 100644 --- a/pod/perlunicode.pod +++ b/pod/perlunicode.pod @@ -737,6 +737,11 @@ The following table is from Unicode 3.2. Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF, the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF. +The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings: +it is technically possible to UTF-8-encode a single code point in different +ways, but that is explicitly forbidden, and the shortest possible encoding +should always be used (and that is what Perl does). + Or, another way to look at it, as bits: Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte diff --git a/utf8.h b/utf8.h index 3787832..0b74517 100644 --- a/utf8.h +++ b/utf8.h @@ -80,6 +80,10 @@ END_EXTERN_C Note the A0..BF in U+0800..U+0FFF, the 80..9F in U+D000...U+D7FF, the 90..BF in U+10000..U+3FFFF, and the 80...8F in U+100000..U+10FFFF. +The "gaps" are caused by legal UTF-8 avoiding non-shortest encodings: +it is technically possible to UTF-8-encode a single code point in different +ways, but that is explicitly forbidden, and the shortest possible encoding +should always be used (and that is what Perl does). */