1 package Unicode::Normalize;
4 unless ("A" eq pack('U', 0x41)) {
5 die "Unicode::Normalize cannot stringify a Unicode code point\n";
14 our $VERSION = '0.23';
15 our $PACKAGE = __PACKAGE__;
21 our @ISA = qw(Exporter DynaLoader);
22 our @EXPORT = qw( NFC NFD NFKC NFKD );
24 normalize decompose reorder compose
25 checkNFD checkNFKD checkNFC checkNFKC check
26 getCanon getCompat getComposite getCombinClass
27 isExclusion isSingleton isNonStDecomp isComp2nd isComp_Ex
28 isNFD_NO isNFC_NO isNFC_MAYBE isNFKD_NO isNFKC_NO isNFKC_MAYBE
31 all => [ @EXPORT, @EXPORT_OK ],
32 normalize => [ @EXPORT, qw/normalize decompose reorder compose/ ],
33 check => [ qw/checkNFD checkNFKD checkNFC checkNFKC check/ ],
36 bootstrap Unicode::Normalize $VERSION;
39 return pack('U*', @_);
43 return unpack('U*', pack('U*').shift);
46 use constant COMPAT => 1;
48 sub NFD ($) { reorder(decompose($_[0])) }
49 sub NFKD ($) { reorder(decompose($_[0], COMPAT)) }
50 sub NFC ($) { compose(reorder(decompose($_[0]))) }
51 sub NFKC ($) { compose(reorder(decompose($_[0], COMPAT))) }
59 $form eq 'D' ? NFD ($str) :
60 $form eq 'C' ? NFC ($str) :
61 $form eq 'KD' ? NFKD($str) :
62 $form eq 'KC' ? NFKC($str) :
63 croak $PACKAGE."::normalize: invalid form name: $form";
72 $form eq 'D' ? checkNFD ($str) :
73 $form eq 'C' ? checkNFC ($str) :
74 $form eq 'KD' ? checkNFKD($str) :
75 $form eq 'KC' ? checkNFKC($str) :
76 croak $PACKAGE."::check: invalid form name: $form";
84 Unicode::Normalize - Unicode Normalization Forms
88 use Unicode::Normalize;
90 $NFD_string = NFD($string); # Normalization Form D
91 $NFC_string = NFC($string); # Normalization Form C
92 $NFKD_string = NFKD($string); # Normalization Form KD
93 $NFKC_string = NFKC($string); # Normalization Form KC
97 use Unicode::Normalize 'normalize';
99 $NFD_string = normalize('D', $string); # Normalization Form D
100 $NFC_string = normalize('C', $string); # Normalization Form C
101 $NFKD_string = normalize('KD', $string); # Normalization Form KD
102 $NFKC_string = normalize('KC', $string); # Normalization Form KC
108 C<$string> is used as a string under character semantics
109 (see F<perlunicode>).
111 C<$codepoint> should be an unsigned integer
112 representing a Unicode code point.
114 Note: Between XS edition and pure Perl edition,
115 interpretation of C<$codepoint> as a decimal number has incompatibility.
116 XS converts C<$codepoint> to an unsigned integer, but pure Perl does not.
117 Do not use a floating point nor a negative sign in C<$codepoint>.
119 =head2 Normalization Forms
123 =item C<$NFD_string = NFD($string)>
125 returns the Normalization Form D (formed by canonical decomposition).
127 =item C<$NFC_string = NFC($string)>
129 returns the Normalization Form C (formed by canonical decomposition
130 followed by canonical composition).
132 =item C<$NFKD_string = NFKD($string)>
134 returns the Normalization Form KD (formed by compatibility decomposition).
136 =item C<$NFKC_string = NFKC($string)>
138 returns the Normalization Form KC (formed by compatibility decomposition
139 followed by B<canonical> composition).
141 =item C<$normalized_string = normalize($form_name, $string)>
143 As C<$form_name>, one of the following names must be given.
145 'C' or 'NFC' for Normalization Form C
146 'D' or 'NFD' for Normalization Form D
147 'KC' or 'NFKC' for Normalization Form KC
148 'KD' or 'NFKD' for Normalization Form KD
152 =head2 Decomposition and Composition
156 =item C<$decomposed_string = decompose($string)>
158 =item C<$decomposed_string = decompose($string, $useCompatMapping)>
160 Decomposes the specified string and returns the result.
162 If the second parameter (a boolean) is omitted or false, decomposes it
163 using the Canonical Decomposition Mapping.
164 If true, decomposes it using the Compatibility Decomposition Mapping.
166 The string returned is not always in NFD/NFKD.
167 Reordering may be required.
169 $NFD_string = reorder(decompose($string)); # eq. to NFD()
170 $NFKD_string = reorder(decompose($string, TRUE)); # eq. to NFKD()
172 =item C<$reordered_string = reorder($string)>
174 Reorders the combining characters and the like in the canonical ordering
175 and returns the result.
177 E.g., when you have a list of NFD/NFKD strings,
178 you can get the concatenated NFD/NFKD string from them, saying
180 $concat_NFD = reorder(join '', @NFD_strings);
181 $concat_NFKD = reorder(join '', @NFKD_strings);
183 =item C<$composed_string = compose($string)>
185 Returns the string where composable pairs are composed.
187 E.g., when you have a NFD/NFKD string,
188 you can get its NFC/NFKC string, saying
190 $NFC_string = compose($NFD_string);
191 $NFKC_string = compose($NFKD_string);
197 (see Annex 8, UAX #15, and F<DerivedNormalizationProps.txt>)
199 The following functions check whether the string is in that normalization form.
201 The result returned will be:
203 YES The string is in that normalization form.
204 NO The string is not in that normalization form.
205 MAYBE Dubious. Maybe yes, maybe no.
209 =item C<$result = checkNFD($string)>
211 returns C<YES> (C<1>) or C<NO> (C<empty string>).
213 =item C<$result = checkNFC($string)>
215 returns C<YES> (C<1>), C<NO> (C<empty string>), or C<MAYBE> (C<undef>).
217 =item C<$result = checkNFKD($string)>
219 returns C<YES> (C<1>) or C<NO> (C<empty string>).
221 =item C<$result = checkNFKC($string)>
223 returns C<YES> (C<1>), C<NO> (C<empty string>), or C<MAYBE> (C<undef>).
225 =item C<$result = check($form_name, $string)>
227 returns C<YES> (C<1>), C<NO> (C<empty string>), or C<MAYBE> (C<undef>).
229 C<$form_name> is alike to that for C<normalize()>.
235 In the cases of NFD and NFKD, the answer must be either C<YES> or C<NO>.
236 The answer C<MAYBE> may be returned in the cases of NFC and NFKC.
238 A MAYBE-NFC/NFKC string should contain at least
239 one combining character or the like.
240 For example, C<COMBINING ACUTE ACCENT> has
241 the MAYBE_NFC/MAYBE_NFKC property.
242 Both C<checkNFC("A\N{COMBINING ACUTE ACCENT}")>
243 and C<checkNFC("B\N{COMBINING ACUTE ACCENT}")> will return C<MAYBE>.
244 C<"A\N{COMBINING ACUTE ACCENT}"> is not in NFC
245 (its NFC is C<"\N{LATIN CAPITAL LETTER A WITH ACUTE}">),
246 while C<"B\N{COMBINING ACUTE ACCENT}"> is in NFC.
248 If you want to check exactly, compare the string with its NFC/NFKC; i.e.,
250 $string eq NFC($string) # more thorough than checkNFC($string)
251 $string eq NFKC($string) # more thorough than checkNFKC($string)
253 =head2 Character Data
255 These functions are interface of character data used internally.
256 If you want only to get Unicode normalization forms, you don't need
261 =item C<$canonical_decomposed = getCanon($codepoint)>
263 If the character of the specified codepoint is canonically
264 decomposable (including Hangul Syllables),
265 returns the B<completely decomposed> string canonically equivalent to it.
267 If it is not decomposable, returns C<undef>.
269 =item C<$compatibility_decomposed = getCompat($codepoint)>
271 If the character of the specified codepoint is compatibility
272 decomposable (including Hangul Syllables),
273 returns the B<completely decomposed> string compatibility equivalent to it.
275 If it is not decomposable, returns C<undef>.
277 =item C<$codepoint_composite = getComposite($codepoint_here, $codepoint_next)>
279 If two characters here and next (as codepoints) are composable
280 (including Hangul Jamo/Syllables and Composition Exclusions),
281 returns the codepoint of the composite.
283 If they are not composable, returns C<undef>.
285 =item C<$combining_class = getCombinClass($codepoint)>
287 Returns the combining class of the character as an integer.
289 =item C<$is_exclusion = isExclusion($codepoint)>
291 Returns a boolean whether the character of the specified codepoint
292 is a composition exclusion.
294 =item C<$is_singleton = isSingleton($codepoint)>
296 Returns a boolean whether the character of the specified codepoint is
299 =item C<$is_non_starter_decomposition = isNonStDecomp($codepoint)>
301 Returns a boolean whether the canonical decomposition
302 of the character of the specified codepoint
303 is a Non-Starter Decomposition.
305 =item C<$may_be_composed_with_prev_char = isComp2nd($codepoint)>
307 Returns a boolean whether the character of the specified codepoint
308 may be composed with the previous one in a certain composition
309 (including Hangul Compositions, but excluding
310 Composition Exclusions and Non-Starter Decompositions).
316 C<NFC>, C<NFD>, C<NFKC>, C<NFKD>: by default.
318 C<normalize> and other some functions: on request.
322 SADAHIRO Tomoyuki, E<lt>SADAHIRO@cpan.orgE<gt>
324 http://homepage1.nifty.com/nomenclator/perl/
326 Copyright(C) 2001-2003, SADAHIRO Tomoyuki. Japan. All rights reserved.
328 This module is free software; you can redistribute it
329 and/or modify it under the same terms as Perl itself.
335 =item http://www.unicode.org/unicode/reports/tr15/
337 Unicode Normalization Forms - UAX #15
339 =item http://www.unicode.org/Public/UNIDATA/DerivedNormalizationProps.txt
341 Derived Normalization Properties