Commit | Line | Data |
5f05dabc |
1 | =head1 NAME |
2 | |
b0c42ed9 |
3 | perllocale - Perl locale handling (internationalization and localization) |
5f05dabc |
4 | |
5 | =head1 DESCRIPTION |
6 | |
7 | Perl supports language-specific notions of data such as "is this a |
68dc0745 |
8 | letter", "what is the uppercase equivalent of this letter", and "which |
14280422 |
9 | of these letters comes first". These are important issues, especially |
10 | for languages other than English - but also for English: it would be |
11 | very naE<iuml>ve to think that C<A-Za-z> defines all the "letters". Perl |
12 | is also aware that some character other than '.' may be preferred as a |
13 | decimal point, and that output date representations may be |
14 | language-specific. The process of making an application take account of |
15 | its users' preferences in such matters is called B<internationalization> |
16 | (often abbreviated as B<i18n>); telling such an application about a |
17 | particular set of preferences is known as B<localization> (B<l10n>). |
18 | |
19 | Perl can understand language-specific data via the standardized (ISO C, |
20 | XPG4, POSIX 1.c) method called "the locale system". The locale system is |
b0c42ed9 |
21 | controlled per application using one pragma, one function call, and |
14280422 |
22 | several environment variables. |
23 | |
24 | B<NOTE>: This feature is new in Perl 5.004, and does not apply unless an |
25 | application specifically requests it - see L<Backward compatibility>. |
e38874e2 |
26 | The one exception is that write() now B<always> uses the current locale |
27 | - see L<"NOTES">. |
5f05dabc |
28 | |
29 | =head1 PREPARING TO USE LOCALES |
30 | |
14280422 |
31 | If Perl applications are to be able to understand and present your data |
32 | correctly according a locale of your choice, B<all> of the following |
5f05dabc |
33 | must be true: |
34 | |
35 | =over 4 |
36 | |
37 | =item * |
38 | |
39 | B<Your operating system must support the locale system>. If it does, |
14280422 |
40 | you should find that the setlocale() function is a documented part of |
5f05dabc |
41 | its C library. |
42 | |
43 | =item * |
44 | |
14280422 |
45 | B<Definitions for the locales which you use must be installed>. You, or |
46 | your system administrator, must make sure that this is the case. The |
47 | available locales, the location in which they are kept, and the manner |
48 | in which they are installed, vary from system to system. Some systems |
4a6725af |
49 | provide only a few, hard-wired, locales, and do not allow more to be |
14280422 |
50 | added; others allow you to add "canned" locales provided by the system |
51 | supplier; still others allow you or the system administrator to define |
52 | and add arbitrary locales. (You may have to ask your supplier to |
53 | provide canned locales which are not delivered with your operating |
54 | system.) Read your system documentation for further illumination. |
5f05dabc |
55 | |
56 | =item * |
57 | |
58 | B<Perl must believe that the locale system is supported>. If it does, |
59 | C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is |
60 | C<define>. |
61 | |
62 | =back |
63 | |
64 | If you want a Perl application to process and present your data |
65 | according to a particular locale, the application code should include |
2ae324a7 |
66 | the S<C<use locale>> pragma (see L<The use locale pragma>) where |
5f05dabc |
67 | appropriate, and B<at least one> of the following must be true: |
68 | |
69 | =over 4 |
70 | |
71 | =item * |
72 | |
14280422 |
73 | B<The locale-determining environment variables (see L<"ENVIRONMENT">) |
74 | must be correctly set up>, either by yourself, or by the person who set |
75 | up your system account, at the time the application is started. |
5f05dabc |
76 | |
77 | =item * |
78 | |
14280422 |
79 | B<The application must set its own locale> using the method described in |
80 | L<The setlocale function>. |
5f05dabc |
81 | |
82 | =back |
83 | |
84 | =head1 USING LOCALES |
85 | |
86 | =head2 The use locale pragma |
87 | |
14280422 |
88 | By default, Perl ignores the current locale. The S<C<use locale>> |
89 | pragma tells Perl to use the current locale for some operations: |
5f05dabc |
90 | |
91 | =over 4 |
92 | |
93 | =item * |
94 | |
14280422 |
95 | B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>) and |
96 | the POSIX string collation functions strcoll() and strxfrm() use |
97 | C<LC_COLLATE>. sort() is also affected if it is used without an |
98 | explicit comparison function because it uses C<cmp> by default. |
99 | |
100 | B<Note:> C<eq> and C<ne> are unaffected by the locale: they always |
101 | perform a byte-by-byte comparison of their scalar operands. What's |
102 | more, if C<cmp> finds that its operands are equal according to the |
103 | collation sequence specified by the current locale, it goes on to |
104 | perform a byte-by-byte comparison, and only returns I<0> (equal) if the |
105 | operands are bit-for-bit identical. If you really want to know whether |
106 | two strings - which C<eq> and C<cmp> may consider different - are equal |
107 | as far as collation in the locale is concerned, see the discussion in |
108 | L<Category LC_COLLATE: Collation>. |
5f05dabc |
109 | |
110 | =item * |
111 | |
14280422 |
112 | B<Regular expressions and case-modification functions> (uc(), lc(), |
113 | ucfirst(), and lcfirst()) use C<LC_CTYPE> |
5f05dabc |
114 | |
115 | =item * |
116 | |
14280422 |
117 | B<The formatting functions> (printf(), sprintf() and write()) use |
5f05dabc |
118 | C<LC_NUMERIC> |
119 | |
120 | =item * |
121 | |
14280422 |
122 | B<The POSIX date formatting function> (strftime()) uses C<LC_TIME>. |
5f05dabc |
123 | |
124 | =back |
125 | |
14280422 |
126 | C<LC_COLLATE>, C<LC_CTYPE>, and so on, are discussed further in L<LOCALE |
127 | CATEGORIES>. |
5f05dabc |
128 | |
b0c42ed9 |
129 | The default behavior returns with S<C<no locale>> or on reaching the |
14280422 |
130 | end of the enclosing block. |
5f05dabc |
131 | |
14280422 |
132 | Note that the string result of any operation that uses locale |
133 | information is tainted, as it is possible for a locale to be |
134 | untrustworthy. See L<"SECURITY">. |
5f05dabc |
135 | |
136 | =head2 The setlocale function |
137 | |
14280422 |
138 | You can switch locales as often as you wish at run time with the |
139 | POSIX::setlocale() function: |
5f05dabc |
140 | |
141 | # This functionality not usable prior to Perl 5.004 |
142 | require 5.004; |
143 | |
144 | # Import locale-handling tool set from POSIX module. |
145 | # This example uses: setlocale -- the function call |
146 | # LC_CTYPE -- explained below |
147 | use POSIX qw(locale_h); |
148 | |
14280422 |
149 | # query and save the old locale |
5f05dabc |
150 | $old_locale = setlocale(LC_CTYPE); |
151 | |
152 | setlocale(LC_CTYPE, "fr_CA.ISO8859-1"); |
153 | # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1" |
154 | |
155 | setlocale(LC_CTYPE, ""); |
156 | # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG |
157 | # environment variables. See below for documentation. |
158 | |
159 | # restore the old locale |
160 | setlocale(LC_CTYPE, $old_locale); |
161 | |
14280422 |
162 | The first argument of setlocale() gives the B<category>, the second the |
163 | B<locale>. The category tells in what aspect of data processing you |
164 | want to apply locale-specific rules. Category names are discussed in |
165 | L<LOCALE CATEGORIES> and L<"ENVIRONMENT">. The locale is the name of a |
166 | collection of customization information corresponding to a particular |
167 | combination of language, country or territory, and codeset. Read on for |
168 | hints on the naming of locales: not all systems name locales as in the |
169 | example. |
170 | |
171 | If no second argument is provided, the function returns a string naming |
172 | the current locale for the category. You can use this value as the |
173 | second argument in a subsequent call to setlocale(). If a second |
5f05dabc |
174 | argument is given and it corresponds to a valid locale, the locale for |
175 | the category is set to that value, and the function returns the |
176 | now-current locale value. You can use this in a subsequent call to |
14280422 |
177 | setlocale(). (In some implementations, the return value may sometimes |
5f05dabc |
178 | differ from the value you gave as the second argument - think of it as |
179 | an alias for the value that you gave.) |
180 | |
181 | As the example shows, if the second argument is an empty string, the |
182 | category's locale is returned to the default specified by the |
183 | corresponding environment variables. Generally, this results in a |
184 | return to the default which was in force when Perl started up: changes |
54310121 |
185 | to the environment made by the application after startup may or may not |
14280422 |
186 | be noticed, depending on the implementation of your system's C library. |
5f05dabc |
187 | |
14280422 |
188 | If the second argument does not correspond to a valid locale, the locale |
189 | for the category is not changed, and the function returns I<undef>. |
5f05dabc |
190 | |
14280422 |
191 | For further information about the categories, consult L<setlocale(3)>. |
3e6e419a |
192 | |
193 | =head2 Finding locales |
194 | |
14280422 |
195 | For the locales available in your system, also consult L<setlocale(3)> |
196 | and see whether it leads you to the list of the available locales |
197 | (search for the I<SEE ALSO> section). If that fails, try the following |
198 | command lines: |
5f05dabc |
199 | |
200 | locale -a |
201 | |
202 | nlsinfo |
203 | |
204 | ls /usr/lib/nls/loc |
205 | |
206 | ls /usr/lib/locale |
207 | |
208 | ls /usr/lib/nls |
209 | |
210 | and see whether they list something resembling these |
211 | |
2bdf8add |
212 | en_US.ISO8859-1 de_DE.ISO8859-1 ru_RU.ISO8859-5 |
213 | en_US de_DE ru_RU |
14280422 |
214 | en de ru |
2bdf8add |
215 | english german russian |
216 | english.iso88591 german.iso88591 russian.iso88595 |
5f05dabc |
217 | |
14280422 |
218 | Sadly, even though the calling interface for setlocale() has been |
3e6e419a |
219 | standardized, the names of the locales and the directories where the |
220 | configuration is, have not. The basic form of the name is |
221 | I<language_country/territory>B<.>I<codeset>, but the latter parts |
222 | after the I<language> are not always present. The I<language> and the |
223 | I<country> are usually from the standards B<ISO 3166> and B<ISO 639>, |
224 | respectively, the two-letter abbreviations for the countries and the |
225 | languages of the world. The I<codeset> part often mentions some B<ISO |
226 | 8859> character set, the Latin codesets. For example the C<ISO |
227 | 8859-1> is the so-called "Western codeset" that can be used to encode |
228 | most of the Western European languages. Again, sadly, as you can see, |
229 | there are several ways to write even the name of that one standard. |
5f05dabc |
230 | |
14280422 |
231 | Two special locales are worth particular mention: "C" and "POSIX". |
232 | Currently these are effectively the same locale: the difference is |
233 | mainly that the first one is defined by the C standard and the second by |
234 | the POSIX standard. What they define is the B<default locale> in which |
235 | every program starts in the absence of locale information in its |
236 | environment. (The default default locale, if you will.) Its language |
237 | is (American) English and its character codeset ASCII. |
5f05dabc |
238 | |
14280422 |
239 | B<NOTE>: Not all systems have the "POSIX" locale (not all systems are |
240 | POSIX-conformant), so use "C" when you need explicitly to specify this |
241 | default locale. |
5f05dabc |
242 | |
3e6e419a |
243 | =head2 LOCALE PROBLEMS |
244 | |
245 | You may meet the following warning message at Perl startup: |
246 | |
247 | perl: warning: Setting locale failed. |
248 | perl: warning: Please check that your locale settings: |
249 | LC_ALL = "En_US", |
250 | LANG = (unset) |
251 | are supported and installed on your system. |
252 | perl: warning: Falling back to the standard locale ("C"). |
253 | |
254 | This means that your locale settings were that LC_ALL equals "En_US" |
255 | and LANG exists but has no value. Perl tried to believe you but it |
256 | could not. Instead Perl gave up and fell back to the "C" locale, the |
257 | default locale that is supposed to work no matter what. This usually |
258 | means either or both of the two problems: either your locale settings |
259 | were wrong, they talk of locales your system has never heard of, or |
260 | that the locale installation in your system has problems, for example |
261 | some system files are broken or missing. For the problems there are |
262 | quick and temporary fixes and more thorough and lasting fixes. |
263 | |
264 | =head2 Temporarily fixing locale problems |
265 | |
266 | The two quickest fixes are either to make Perl be silent about any |
267 | locale inconsistencies or to run Perl under the default locale "C". |
268 | |
269 | Perl's moaning about locale problems can be silenced by setting the |
270 | environment variable PERL_BADLANG to a non-zero value, for example |
271 | "1". This method really just sweeps the problem under the carpet: you |
272 | tell Perl to shut up even when Perl sees that something is wrong. Do |
273 | not be surprised if later something locale-dependent works funny. |
274 | |
275 | Perl can be run under the "C" locale by setting the environment |
276 | variable LC_ALL to "C". This method is perhaps a bit more civilised |
277 | than the PERL_BADLANG one but please note that setting the LC_ALL (or |
278 | the other locale variables) may affect also other programs, not just |
279 | Perl. Especially external programs run from within Perl will see |
280 | these changes. If you make the new settings permanent (read on), all |
281 | the programs you run will see the changes. See L<ENVIRONMENT> for for |
282 | the full list of all the environment variables and L<USING LOCALES> |
283 | for their effects in Perl. The effects in other programs are quite |
284 | easily deducible: for example the variable LC_COLLATE may well affect |
285 | your "sort" program (or whatever the program that arranges `records' |
286 | alphabetically in your system is called). |
287 | |
288 | You can first try out changing these variables temporarily and if the |
289 | new settings seem to help then put the settings into the startup files |
290 | of your environment. Please consult your local documentation for the |
291 | exact details but very shortly for UNIXish systems: in Bourneish |
292 | shells (sh, ksh, bash, zsh) for example |
293 | |
294 | LC_ALL=en_US.ISO8859-1 |
295 | export LC_ALL |
296 | |
297 | We assume here that we saw with the above discussed commands the |
298 | locale "en_US.ISO8859-1" and decided to try that instead of the above |
299 | faulty locale "En_US" -- and in Cshish shells (csh, tcsh) |
300 | |
301 | setenv LC_ALL en_US.ISO8859-1 |
302 | |
303 | If you do not know what shell you have, please consult your local |
304 | helpdesk or the equivalent. |
305 | |
306 | =head2 Permanently fixing locale problems |
307 | |
308 | Then the slower but better fixes: the misconfiguration of your own |
309 | environment variables you may be able to fix yourself; the |
310 | mis(sing)configuration of the whole system's locales usually requires |
311 | the help of your friendly system administrator. |
312 | |
313 | First, see earlier in this document about L<Finding locales>. That |
314 | tells how you can find which locales really are supported and more |
315 | importantly, installed, in your system. In our example error message |
316 | the environment variables affecting the locale are listed in the order |
317 | of decreasing importance and unset variables do not matter, therefore |
318 | in the above error message the LC_ALL being "En_US" must have been the |
319 | bad choice. Always try fixing first the locale settings listed first. |
320 | |
321 | Second, if you see with the listed commands something B<exactly> (for |
322 | example prefix matches do not count and case usually matters) like |
323 | "En_US" (without the quotes), then you should be okay because you are |
324 | using a locale name that should be installed and available in your |
325 | system. In this case skip forward to L<Fixing the system locale |
326 | configuration>. |
327 | |
328 | =head2 Permantently fixing your locale configuration |
329 | |
330 | This is the case when for example you see |
331 | |
332 | perl: warning: Please check that your locale settings: |
333 | LC_ALL = "En_US", |
334 | LANG = (unset) |
335 | are supported and installed on your system. |
336 | |
337 | but then cannot see that "En_US" listed by the above-mentioned |
338 | commands. You may see things like "en_US.ISO8859-1" but that is not |
339 | the same thing. In this case you might try running under a locale |
340 | that you could list and somehow matches with what you tried. The |
341 | rules for matching locale names are a bit vague because |
342 | standardisation is weak in this area. See again the L<Finding |
343 | locales> about the general rules. |
344 | |
345 | =head2 Permanently fixing the system locale configuration |
346 | |
347 | Please contact your system administrator and tell her the exact error |
348 | message you get and ask her to read this same documentation you are |
349 | now reading. She should be able to check whether there is something |
350 | wrong with the locale configuration of the system. The L<Finding |
351 | locales> section is unfortunately a bit vague about the exact commands |
352 | and places because these things are not that standardised. |
353 | |
5f05dabc |
354 | =head2 The localeconv function |
355 | |
14280422 |
356 | The POSIX::localeconv() function allows you to get particulars of the |
357 | locale-dependent numeric formatting information specified by the current |
358 | C<LC_NUMERIC> and C<LC_MONETARY> locales. (If you just want the name of |
359 | the current locale for a particular category, use POSIX::setlocale() |
360 | with a single parameter - see L<The setlocale function>.) |
5f05dabc |
361 | |
362 | use POSIX qw(locale_h); |
5f05dabc |
363 | |
364 | # Get a reference to a hash of locale-dependent info |
365 | $locale_values = localeconv(); |
366 | |
367 | # Output sorted list of the values |
368 | for (sort keys %$locale_values) { |
14280422 |
369 | printf "%-20s = %s\n", $_, $locale_values->{$_} |
5f05dabc |
370 | } |
371 | |
14280422 |
372 | localeconv() takes no arguments, and returns B<a reference to> a hash. |
373 | The keys of this hash are formatting variable names such as |
374 | C<decimal_point> and C<thousands_sep>; the values are the corresponding |
375 | values. See L<POSIX (3)/localeconv> for a longer example, which lists |
376 | all the categories an implementation might be expected to provide; some |
377 | provide more and others fewer, however. Note that you don't need C<use |
378 | locale>: as a function with the job of querying the locale, localeconv() |
379 | always observes the current locale. |
5f05dabc |
380 | |
381 | Here's a simple-minded example program which rewrites its command line |
382 | parameters as integers formatted correctly in the current locale: |
383 | |
384 | # See comments in previous example |
385 | require 5.004; |
386 | use POSIX qw(locale_h); |
5f05dabc |
387 | |
388 | # Get some of locale's numeric formatting parameters |
389 | my ($thousands_sep, $grouping) = |
14280422 |
390 | @{localeconv()}{'thousands_sep', 'grouping'}; |
5f05dabc |
391 | |
392 | # Apply defaults if values are missing |
393 | $thousands_sep = ',' unless $thousands_sep; |
394 | $grouping = 3 unless $grouping; |
395 | |
396 | # Format command line params for current locale |
14280422 |
397 | for (@ARGV) { |
398 | $_ = int; # Chop non-integer part |
5f05dabc |
399 | 1 while |
14280422 |
400 | s/(\d)(\d{$grouping}($|$thousands_sep))/$1$thousands_sep$2/; |
401 | print "$_"; |
5f05dabc |
402 | } |
403 | print "\n"; |
404 | |
5f05dabc |
405 | =head1 LOCALE CATEGORIES |
406 | |
14280422 |
407 | The subsections which follow describe basic locale categories. As well |
5f05dabc |
408 | as these, there are some combination categories which allow the |
14280422 |
409 | manipulation of more than one basic category at a time. See |
410 | L<"ENVIRONMENT"> for a discussion of these. |
5f05dabc |
411 | |
412 | =head2 Category LC_COLLATE: Collation |
413 | |
14280422 |
414 | When in the scope of S<C<use locale>>, Perl looks to the C<LC_COLLATE> |
5f05dabc |
415 | environment variable to determine the application's notions on the |
14280422 |
416 | collation (ordering) of characters. ('b' follows 'a' in Latin |
417 | alphabets, but where do 'E<aacute>' and 'E<aring>' belong?) |
5f05dabc |
418 | |
419 | Here is a code snippet that will tell you what are the alphanumeric |
420 | characters in the current locale, in the locale order: |
421 | |
422 | use locale; |
423 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
424 | |
14280422 |
425 | Compare this with the characters that you see and their order if you |
426 | state explicitly that the locale should be ignored: |
5f05dabc |
427 | |
428 | no locale; |
429 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
430 | |
431 | This machine-native collation (which is what you get unless S<C<use |
432 | locale>> has appeared earlier in the same block) must be used for |
433 | sorting raw binary data, whereas the locale-dependent collation of the |
b0c42ed9 |
434 | first example is useful for natural text. |
5f05dabc |
435 | |
14280422 |
436 | As noted in L<USING LOCALES>, C<cmp> compares according to the current |
437 | collation locale when C<use locale> is in effect, but falls back to a |
438 | byte-by-byte comparison for strings which the locale says are equal. You |
439 | can use POSIX::strcoll() if you don't want this fall-back: |
440 | |
441 | use POSIX qw(strcoll); |
442 | $equal_in_locale = |
443 | !strcoll("space and case ignored", "SpaceAndCaseIgnored"); |
444 | |
445 | $equal_in_locale will be true if the collation locale specifies a |
446 | dictionary-like ordering which ignores space characters completely, and |
9e3a2af8 |
447 | which folds case. |
14280422 |
448 | |
449 | If you have a single string which you want to check for "equality in |
450 | locale" against several others, you might think you could gain a little |
451 | efficiency by using POSIX::strxfrm() in conjunction with C<eq>: |
452 | |
453 | use POSIX qw(strxfrm); |
454 | $xfrm_string = strxfrm("Mixed-case string"); |
455 | print "locale collation ignores spaces\n" |
456 | if $xfrm_string eq strxfrm("Mixed-casestring"); |
457 | print "locale collation ignores hyphens\n" |
458 | if $xfrm_string eq strxfrm("Mixedcase string"); |
459 | print "locale collation ignores case\n" |
460 | if $xfrm_string eq strxfrm("mixed-case string"); |
461 | |
462 | strxfrm() takes a string and maps it into a transformed string for use |
463 | in byte-by-byte comparisons against other transformed strings during |
464 | collation. "Under the hood", locale-affected Perl comparison operators |
465 | call strxfrm() for both their operands, then do a byte-by-byte |
466 | comparison of the transformed strings. By calling strxfrm() explicitly, |
467 | and using a non locale-affected comparison, the example attempts to save |
468 | a couple of transformations. In fact, it doesn't save anything: Perl |
2ae324a7 |
469 | magic (see L<perlguts/Magic Variables>) creates the transformed version of a |
14280422 |
470 | string the first time it's needed in a comparison, then keeps it around |
471 | in case it's needed again. An example rewritten the easy way with |
e38874e2 |
472 | C<cmp> runs just about as fast. It also copes with null characters |
14280422 |
473 | embedded in strings; if you call strxfrm() directly, it treats the first |
e38874e2 |
474 | null it finds as a terminator. And don't expect the transformed strings |
475 | it produces to be portable across systems - or even from one revision |
476 | of your operating system to the next. In short, don't call strxfrm() |
477 | directly: let Perl do it for you. |
14280422 |
478 | |
479 | Note: C<use locale> isn't shown in some of these examples, as it isn't |
480 | needed: strcoll() and strxfrm() exist only to generate locale-dependent |
481 | results, and so always obey the current C<LC_COLLATE> locale. |
5f05dabc |
482 | |
483 | =head2 Category LC_CTYPE: Character Types |
484 | |
485 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_CTYPE> locale |
14280422 |
486 | setting. This controls the application's notion of which characters are |
487 | alphabetic. This affects Perl's C<\w> regular expression metanotation, |
488 | which stands for alphanumeric characters - that is, alphabetic and |
489 | numeric characters. (Consult L<perlre> for more information about |
490 | regular expressions.) Thanks to C<LC_CTYPE>, depending on your locale |
491 | setting, characters like 'E<aelig>', 'E<eth>', 'E<szlig>', and |
492 | 'E<oslash>' may be understood as C<\w> characters. |
5f05dabc |
493 | |
2c268ad5 |
494 | The C<LC_CTYPE> locale also provides the map used in transliterating |
68dc0745 |
495 | characters between lower and uppercase. This affects the case-mapping |
e38874e2 |
496 | functions - lc(), lcfirst, uc() and ucfirst(); case-mapping |
497 | interpolation with C<\l>, C<\L>, C<\u> or <\U> in double-quoted strings |
498 | and in C<s///> substitutions; and case-independent regular expression |
499 | pattern matching using the C<i> modifier. |
500 | |
501 | Finally, C<LC_CTYPE> affects the POSIX character-class test functions - |
14280422 |
502 | isalpha(), islower() and so on. For example, if you move from the "C" |
503 | locale to a 7-bit Scandinavian one, you may find - possibly to your |
504 | surprise - that "|" moves from the ispunct() class to isalpha(). |
5f05dabc |
505 | |
14280422 |
506 | B<Note:> A broken or malicious C<LC_CTYPE> locale definition may result |
507 | in clearly ineligible characters being considered to be alphanumeric by |
508 | your application. For strict matching of (unaccented) letters and |
509 | digits - for example, in command strings - locale-aware applications |
510 | should use C<\w> inside a C<no locale> block. See L<"SECURITY">. |
5f05dabc |
511 | |
512 | =head2 Category LC_NUMERIC: Numeric Formatting |
513 | |
514 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_NUMERIC> |
14280422 |
515 | locale information, which controls application's idea of how numbers |
516 | should be formatted for human readability by the printf(), sprintf(), |
517 | and write() functions. String to numeric conversion by the |
518 | POSIX::strtod() function is also affected. In most implementations the |
519 | only effect is to change the character used for the decimal point - |
520 | perhaps from '.' to ',': these functions aren't aware of such niceties |
521 | as thousands separation and so on. (See L<The localeconv function> if |
522 | you care about these things.) |
523 | |
524 | Note that output produced by print() is B<never> affected by the |
5f05dabc |
525 | current locale: it is independent of whether C<use locale> or C<no |
14280422 |
526 | locale> is in effect, and corresponds to what you'd get from printf() |
5f05dabc |
527 | in the "C" locale. The same is true for Perl's internal conversions |
528 | between numeric and string formats: |
529 | |
530 | use POSIX qw(strtod); |
531 | use locale; |
14280422 |
532 | |
5f05dabc |
533 | $n = 5/2; # Assign numeric 2.5 to $n |
534 | |
535 | $a = " $n"; # Locale-independent conversion to string |
536 | |
537 | print "half five is $n\n"; # Locale-independent output |
538 | |
539 | printf "half five is %g\n", $n; # Locale-dependent output |
540 | |
14280422 |
541 | print "DECIMAL POINT IS COMMA\n" |
542 | if $n == (strtod("2,5"))[0]; # Locale-dependent conversion |
5f05dabc |
543 | |
544 | =head2 Category LC_MONETARY: Formatting of monetary amounts |
545 | |
14280422 |
546 | The C standard defines the C<LC_MONETARY> category, but no function that |
547 | is affected by its contents. (Those with experience of standards |
b0c42ed9 |
548 | committees will recognize that the working group decided to punt on the |
14280422 |
549 | issue.) Consequently, Perl takes no notice of it. If you really want |
550 | to use C<LC_MONETARY>, you can query its contents - see L<The localeconv |
551 | function> - and use the information that it returns in your |
b0c42ed9 |
552 | application's own formatting of currency amounts. However, you may well |
14280422 |
553 | find that the information, though voluminous and complex, does not quite |
554 | meet your requirements: currency formatting is a hard nut to crack. |
5f05dabc |
555 | |
556 | =head2 LC_TIME |
557 | |
14280422 |
558 | The output produced by POSIX::strftime(), which builds a formatted |
5f05dabc |
559 | human-readable date/time string, is affected by the current C<LC_TIME> |
560 | locale. Thus, in a French locale, the output produced by the C<%B> |
561 | format element (full month name) for the first month of the year would |
562 | be "janvier". Here's how to get a list of the long month names in the |
563 | current locale: |
564 | |
565 | use POSIX qw(strftime); |
14280422 |
566 | for (0..11) { |
567 | $long_month_name[$_] = |
568 | strftime("%B", 0, 0, 0, 1, $_, 96); |
5f05dabc |
569 | } |
570 | |
14280422 |
571 | Note: C<use locale> isn't needed in this example: as a function which |
572 | exists only to generate locale-dependent results, strftime() always |
573 | obeys the current C<LC_TIME> locale. |
5f05dabc |
574 | |
575 | =head2 Other categories |
576 | |
577 | The remaining locale category, C<LC_MESSAGES> (possibly supplemented by |
578 | others in particular implementations) is not currently used by Perl - |
b0c42ed9 |
579 | except possibly to affect the behavior of library functions called by |
14280422 |
580 | extensions which are not part of the standard Perl distribution. |
581 | |
582 | =head1 SECURITY |
583 | |
584 | While the main discussion of Perl security issues can be found in |
585 | L<perlsec>, a discussion of Perl's locale handling would be incomplete |
586 | if it did not draw your attention to locale-dependent security issues. |
587 | Locales - particularly on systems which allow unprivileged users to |
588 | build their own locales - are untrustworthy. A malicious (or just plain |
589 | broken) locale can make a locale-aware application give unexpected |
590 | results. Here are a few possibilities: |
591 | |
592 | =over 4 |
593 | |
594 | =item * |
595 | |
596 | Regular expression checks for safe file names or mail addresses using |
597 | C<\w> may be spoofed by an C<LC_CTYPE> locale which claims that |
598 | characters such as "E<gt>" and "|" are alphanumeric. |
599 | |
600 | =item * |
601 | |
e38874e2 |
602 | String interpolation with case-mapping, as in, say, C<$dest = |
603 | "C:\U$name.$ext">, may produce dangerous results if a bogus LC_CTYPE |
604 | case-mapping table is in effect. |
605 | |
606 | =item * |
607 | |
14280422 |
608 | If the decimal point character in the C<LC_NUMERIC> locale is |
609 | surreptitiously changed from a dot to a comma, C<sprintf("%g", |
610 | 0.123456e3)> produces a string result of "123,456". Many people would |
611 | interpret this as one hundred and twenty-three thousand, four hundred |
612 | and fifty-six. |
613 | |
614 | =item * |
615 | |
616 | A sneaky C<LC_COLLATE> locale could result in the names of students with |
617 | "D" grades appearing ahead of those with "A"s. |
618 | |
619 | =item * |
620 | |
621 | An application which takes the trouble to use the information in |
622 | C<LC_MONETARY> may format debits as if they were credits and vice versa |
623 | if that locale has been subverted. Or it make may make payments in US |
624 | dollars instead of Hong Kong dollars. |
625 | |
626 | =item * |
627 | |
628 | The date and day names in dates formatted by strftime() could be |
629 | manipulated to advantage by a malicious user able to subvert the |
630 | C<LC_DATE> locale. ("Look - it says I wasn't in the building on |
631 | Sunday.") |
632 | |
633 | =back |
634 | |
635 | Such dangers are not peculiar to the locale system: any aspect of an |
636 | application's environment which may maliciously be modified presents |
637 | similar challenges. Similarly, they are not specific to Perl: any |
638 | programming language which allows you to write programs which take |
639 | account of their environment exposes you to these issues. |
640 | |
641 | Perl cannot protect you from all of the possibilities shown in the |
642 | examples - there is no substitute for your own vigilance - but, when |
643 | C<use locale> is in effect, Perl uses the tainting mechanism (see |
644 | L<perlsec>) to mark string results which become locale-dependent, and |
645 | which may be untrustworthy in consequence. Here is a summary of the |
b0c42ed9 |
646 | tainting behavior of operators and functions which may be affected by |
14280422 |
647 | the locale: |
648 | |
649 | =over 4 |
650 | |
651 | =item B<Comparison operators> (C<lt>, C<le>, C<ge>, C<gt> and C<cmp>): |
652 | |
653 | Scalar true/false (or less/equal/greater) result is never tainted. |
654 | |
e38874e2 |
655 | =item B<Case-mapping interpolation> (with C<\l>, C<\L>, C<\u> or <\U>) |
656 | |
657 | Result string containing interpolated material is tainted if |
658 | C<use locale> is in effect. |
659 | |
14280422 |
660 | =item B<Matching operator> (C<m//>): |
661 | |
662 | Scalar true/false result never tainted. |
663 | |
664 | Subpatterns, either delivered as an array-context result, or as $1 etc. |
665 | are tainted if C<use locale> is in effect, and the subpattern regular |
e38874e2 |
666 | expression contains C<\w> (to match an alphanumeric character), C<\W> |
667 | (non-alphanumeric character), C<\s> (white-space character), or C<\S> |
668 | (non white-space character). The matched pattern variable, $&, $` |
669 | (pre-match), $' (post-match), and $+ (last match) are also tainted if |
670 | C<use locale> is in effect and the regular expression contains C<\w>, |
671 | C<\W>, C<\s>, or C<\S>. |
14280422 |
672 | |
673 | =item B<Substitution operator> (C<s///>): |
674 | |
e38874e2 |
675 | Has the same behavior as the match operator. Also, the left |
676 | operand of C<=~> becomes tainted when C<use locale> in effect, |
677 | if it is modified as a result of a substitution based on a regular |
678 | expression match involving C<\w>, C<\W>, C<\s>, or C<\S>; or of |
679 | case-mapping with C<\l>, C<\L>,C<\u> or <\U>. |
14280422 |
680 | |
681 | =item B<In-memory formatting function> (sprintf()): |
682 | |
683 | Result is tainted if "use locale" is in effect. |
684 | |
685 | =item B<Output formatting functions> (printf() and write()): |
686 | |
687 | Success/failure result is never tainted. |
688 | |
689 | =item B<Case-mapping functions> (lc(), lcfirst(), uc(), ucfirst()): |
690 | |
691 | Results are tainted if C<use locale> is in effect. |
692 | |
693 | =item B<POSIX locale-dependent functions> (localeconv(), strcoll(), |
694 | strftime(), strxfrm()): |
695 | |
696 | Results are never tainted. |
697 | |
698 | =item B<POSIX character class tests> (isalnum(), isalpha(), isdigit(), |
699 | isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(), |
700 | isxdigit()): |
701 | |
702 | True/false results are never tainted. |
703 | |
704 | =back |
705 | |
706 | Three examples illustrate locale-dependent tainting. |
707 | The first program, which ignores its locale, won't run: a value taken |
54310121 |
708 | directly from the command line may not be used to name an output file |
14280422 |
709 | when taint checks are enabled. |
710 | |
711 | #/usr/local/bin/perl -T |
712 | # Run with taint checking |
713 | |
54310121 |
714 | # Command line sanity check omitted... |
14280422 |
715 | $tainted_output_file = shift; |
716 | |
717 | open(F, ">$tainted_output_file") |
718 | or warn "Open of $untainted_output_file failed: $!\n"; |
719 | |
720 | The program can be made to run by "laundering" the tainted value through |
721 | a regular expression: the second example - which still ignores locale |
54310121 |
722 | information - runs, creating the file named on its command line |
14280422 |
723 | if it can. |
724 | |
725 | #/usr/local/bin/perl -T |
726 | |
727 | $tainted_output_file = shift; |
728 | $tainted_output_file =~ m%[\w/]+%; |
729 | $untainted_output_file = $&; |
730 | |
731 | open(F, ">$untainted_output_file") |
732 | or warn "Open of $untainted_output_file failed: $!\n"; |
733 | |
734 | Compare this with a very similar program which is locale-aware: |
735 | |
736 | #/usr/local/bin/perl -T |
737 | |
738 | $tainted_output_file = shift; |
739 | use locale; |
740 | $tainted_output_file =~ m%[\w/]+%; |
741 | $localized_output_file = $&; |
742 | |
743 | open(F, ">$localized_output_file") |
744 | or warn "Open of $localized_output_file failed: $!\n"; |
745 | |
746 | This third program fails to run because $& is tainted: it is the result |
747 | of a match involving C<\w> when C<use locale> is in effect. |
5f05dabc |
748 | |
749 | =head1 ENVIRONMENT |
750 | |
751 | =over 12 |
752 | |
753 | =item PERL_BADLANG |
754 | |
14280422 |
755 | A string that can suppress Perl's warning about failed locale settings |
54310121 |
756 | at startup. Failure can occur if the locale support in the operating |
14280422 |
757 | system is lacking (broken) is some way - or if you mistyped the name of |
758 | a locale when you set up your environment. If this environment variable |
759 | is absent, or has a value which does not evaluate to integer zero - that |
760 | is "0" or "" - Perl will complain about locale setting failures. |
5f05dabc |
761 | |
14280422 |
762 | B<NOTE>: PERL_BADLANG only gives you a way to hide the warning message. |
763 | The message tells about some problem in your system's locale support, |
764 | and you should investigate what the problem is. |
5f05dabc |
765 | |
766 | =back |
767 | |
768 | The following environment variables are not specific to Perl: They are |
14280422 |
769 | part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale() method |
770 | for controlling an application's opinion on data. |
5f05dabc |
771 | |
772 | =over 12 |
773 | |
774 | =item LC_ALL |
775 | |
776 | C<LC_ALL> is the "override-all" locale environment variable. If it is |
777 | set, it overrides all the rest of the locale environment variables. |
778 | |
779 | =item LC_CTYPE |
780 | |
781 | In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type |
782 | locale. In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG> |
783 | chooses the character type locale. |
784 | |
785 | =item LC_COLLATE |
786 | |
14280422 |
787 | In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation |
788 | (sorting) locale. In the absence of both C<LC_ALL> and C<LC_COLLATE>, |
789 | C<LANG> chooses the collation locale. |
5f05dabc |
790 | |
791 | =item LC_MONETARY |
792 | |
14280422 |
793 | In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the monetary |
794 | formatting locale. In the absence of both C<LC_ALL> and C<LC_MONETARY>, |
795 | C<LANG> chooses the monetary formatting locale. |
5f05dabc |
796 | |
797 | =item LC_NUMERIC |
798 | |
799 | In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format |
800 | locale. In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG> |
801 | chooses the numeric format. |
802 | |
803 | =item LC_TIME |
804 | |
14280422 |
805 | In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time |
806 | formatting locale. In the absence of both C<LC_ALL> and C<LC_TIME>, |
807 | C<LANG> chooses the date and time formatting locale. |
5f05dabc |
808 | |
809 | =item LANG |
810 | |
14280422 |
811 | C<LANG> is the "catch-all" locale environment variable. If it is set, it |
812 | is used as the last resort after the overall C<LC_ALL> and the |
5f05dabc |
813 | category-specific C<LC_...>. |
814 | |
815 | =back |
816 | |
817 | =head1 NOTES |
818 | |
819 | =head2 Backward compatibility |
820 | |
b0c42ed9 |
821 | Versions of Perl prior to 5.004 B<mostly> ignored locale information, |
822 | generally behaving as if something similar to the C<"C"> locale (see |
823 | L<The setlocale function>) was always in force, even if the program |
5f05dabc |
824 | environment suggested otherwise. By default, Perl still behaves this |
825 | way so as to maintain backward compatibility. If you want a Perl |
b0c42ed9 |
826 | application to pay attention to locale information, you B<must> use |
2ae324a7 |
827 | the S<C<use locale>> pragma (see L<The use locale Pragma>) to |
b0c42ed9 |
828 | instruct it to do so. |
829 | |
830 | Versions of Perl from 5.002 to 5.003 did use the C<LC_CTYPE> |
831 | information if that was available, that is, C<\w> did understand what |
832 | are the letters according to the locale environment variables. |
833 | The problem was that the user had no control over the feature: |
834 | if the C library supported locales, Perl used them. |
835 | |
836 | =head2 I18N:Collate obsolete |
837 | |
838 | In versions of Perl prior to 5.004 per-locale collation was possible |
839 | using the C<I18N::Collate> library module. This module is now mildly |
840 | obsolete and should be avoided in new applications. The C<LC_COLLATE> |
841 | functionality is now integrated into the Perl core language: One can |
842 | use locale-specific scalar data completely normally with C<use locale>, |
843 | so there is no longer any need to juggle with the scalar references of |
844 | C<I18N::Collate>. |
5f05dabc |
845 | |
14280422 |
846 | =head2 Sort speed and memory use impacts |
5f05dabc |
847 | |
848 | Comparing and sorting by locale is usually slower than the default |
14280422 |
849 | sorting; slow-downs of two to four times have been observed. It will |
850 | also consume more memory: once a Perl scalar variable has participated |
851 | in any string comparison or sorting operation obeying the locale |
852 | collation rules, it will take 3-15 times more memory than before. (The |
853 | exact multiplier depends on the string's contents, the operating system |
854 | and the locale.) These downsides are dictated more by the operating |
855 | system's implementation of the locale system than by Perl. |
5f05dabc |
856 | |
e38874e2 |
857 | =head2 write() and LC_NUMERIC |
858 | |
859 | Formats are the only part of Perl which unconditionally use information |
860 | from a program's locale; if a program's environment specifies an |
861 | LC_NUMERIC locale, it is always used to specify the decimal point |
862 | character in formatted output. Formatted output cannot be controlled by |
863 | C<use locale> because the pragma is tied to the block structure of the |
864 | program, and, for historical reasons, formats exist outside that block |
865 | structure. |
866 | |
5f05dabc |
867 | =head2 Freely available locale definitions |
868 | |
869 | There is a large collection of locale definitions at |
14280422 |
870 | C<ftp://dkuug.dk/i18n/WG15-collection>. You should be aware that it is |
871 | unsupported, and is not claimed to be fit for any purpose. If your |
872 | system allows the installation of arbitrary locales, you may find the |
873 | definitions useful as they are, or as a basis for the development of |
874 | your own locales. |
5f05dabc |
875 | |
14280422 |
876 | =head2 I18n and l10n |
5f05dabc |
877 | |
b0c42ed9 |
878 | "Internationalization" is often abbreviated as B<i18n> because its first |
879 | and last letters are separated by eighteen others. (You may guess why |
880 | the internalin ... internaliti ... i18n tends to get abbreviated.) In |
881 | the same way, "localization" is often abbreviated to B<l10n>. |
14280422 |
882 | |
883 | =head2 An imperfect standard |
884 | |
885 | Internationalization, as defined in the C and POSIX standards, can be |
886 | criticized as incomplete, ungainly, and having too large a granularity. |
887 | (Locales apply to a whole process, when it would arguably be more useful |
888 | to have them apply to a single thread, window group, or whatever.) They |
889 | also have a tendency, like standards groups, to divide the world into |
890 | nations, when we all know that the world can equally well be divided |
891 | into bankers, bikers, gamers, and so on. But, for now, it's the only |
892 | standard we've got. This may be construed as a bug. |
5f05dabc |
893 | |
894 | =head1 BUGS |
895 | |
896 | =head2 Broken systems |
897 | |
2bdf8add |
898 | In certain system environments the operating system's locale support |
899 | is broken and cannot be fixed or used by Perl. Such deficiencies can |
900 | and will result in mysterious hangs and/or Perl core dumps when the |
901 | C<use locale> is in effect. When confronted with such a system, |
9607fc9c |
902 | please report in excruciating detail to <F<perlbug@perl.com>>, and |
2bdf8add |
903 | complain to your vendor: maybe some bug fixes exist for these problems |
904 | in your operating system. Sometimes such bug fixes are called an |
905 | operating system upgrade. |
5f05dabc |
906 | |
907 | =head1 SEE ALSO |
908 | |
909 | L<POSIX (3)/isalnum>, L<POSIX (3)/isalpha>, L<POSIX (3)/isdigit>, |
910 | L<POSIX (3)/isgraph>, L<POSIX (3)/islower>, L<POSIX (3)/isprint>, |
911 | L<POSIX (3)/ispunct>, L<POSIX (3)/isspace>, L<POSIX (3)/isupper>, |
912 | L<POSIX (3)/isxdigit>, L<POSIX (3)/localeconv>, L<POSIX (3)/setlocale>, |
14280422 |
913 | L<POSIX (3)/strcoll>, L<POSIX (3)/strftime>, L<POSIX (3)/strtod>, |
914 | L<POSIX (3)/strxfrm> |
5f05dabc |
915 | |
916 | =head1 HISTORY |
917 | |
b0c42ed9 |
918 | Jarkko Hietaniemi's original F<perli18n.pod> heavily hacked by Dominic |
14280422 |
919 | Dunlop, assisted by the perl5-porters. |
5f05dabc |
920 | |
3e6e419a |
921 | Last update: Mon Nov 17 22:48:48 EET 1997 |