Commit | Line | Data |
5f05dabc |
1 | =head1 NAME |
2 | |
b0c42ed9 |
3 | perllocale - Perl locale handling (internationalization and localization) |
5f05dabc |
4 | |
5 | =head1 DESCRIPTION |
6 | |
7 | Perl supports language-specific notions of data such as "is this a |
14280422 |
8 | letter", "what is the upper-case equivalent of this letter", and "which |
9 | of these letters comes first". These are important issues, especially |
10 | for languages other than English - but also for English: it would be |
11 | very naE<iuml>ve to think that C<A-Za-z> defines all the "letters". Perl |
12 | is also aware that some character other than '.' may be preferred as a |
13 | decimal point, and that output date representations may be |
14 | language-specific. The process of making an application take account of |
15 | its users' preferences in such matters is called B<internationalization> |
16 | (often abbreviated as B<i18n>); telling such an application about a |
17 | particular set of preferences is known as B<localization> (B<l10n>). |
18 | |
19 | Perl can understand language-specific data via the standardized (ISO C, |
20 | XPG4, POSIX 1.c) method called "the locale system". The locale system is |
b0c42ed9 |
21 | controlled per application using one pragma, one function call, and |
14280422 |
22 | several environment variables. |
23 | |
24 | B<NOTE>: This feature is new in Perl 5.004, and does not apply unless an |
25 | application specifically requests it - see L<Backward compatibility>. |
5f05dabc |
26 | |
27 | =head1 PREPARING TO USE LOCALES |
28 | |
14280422 |
29 | If Perl applications are to be able to understand and present your data |
30 | correctly according a locale of your choice, B<all> of the following |
5f05dabc |
31 | must be true: |
32 | |
33 | =over 4 |
34 | |
35 | =item * |
36 | |
37 | B<Your operating system must support the locale system>. If it does, |
14280422 |
38 | you should find that the setlocale() function is a documented part of |
5f05dabc |
39 | its C library. |
40 | |
41 | =item * |
42 | |
14280422 |
43 | B<Definitions for the locales which you use must be installed>. You, or |
44 | your system administrator, must make sure that this is the case. The |
45 | available locales, the location in which they are kept, and the manner |
46 | in which they are installed, vary from system to system. Some systems |
47 | provide only a few, hard-wired, locales, and do not allow more to be |
48 | added; others allow you to add "canned" locales provided by the system |
49 | supplier; still others allow you or the system administrator to define |
50 | and add arbitrary locales. (You may have to ask your supplier to |
51 | provide canned locales which are not delivered with your operating |
52 | system.) Read your system documentation for further illumination. |
5f05dabc |
53 | |
54 | =item * |
55 | |
56 | B<Perl must believe that the locale system is supported>. If it does, |
57 | C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is |
58 | C<define>. |
59 | |
60 | =back |
61 | |
62 | If you want a Perl application to process and present your data |
63 | according to a particular locale, the application code should include |
14280422 |
64 | the S<C<use locale>> pragma (see L<The use locale Pragma>) where |
5f05dabc |
65 | appropriate, and B<at least one> of the following must be true: |
66 | |
67 | =over 4 |
68 | |
69 | =item * |
70 | |
14280422 |
71 | B<The locale-determining environment variables (see L<"ENVIRONMENT">) |
72 | must be correctly set up>, either by yourself, or by the person who set |
73 | up your system account, at the time the application is started. |
5f05dabc |
74 | |
75 | =item * |
76 | |
14280422 |
77 | B<The application must set its own locale> using the method described in |
78 | L<The setlocale function>. |
5f05dabc |
79 | |
80 | =back |
81 | |
82 | =head1 USING LOCALES |
83 | |
84 | =head2 The use locale pragma |
85 | |
14280422 |
86 | By default, Perl ignores the current locale. The S<C<use locale>> |
87 | pragma tells Perl to use the current locale for some operations: |
5f05dabc |
88 | |
89 | =over 4 |
90 | |
91 | =item * |
92 | |
14280422 |
93 | B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>) and |
94 | the POSIX string collation functions strcoll() and strxfrm() use |
95 | C<LC_COLLATE>. sort() is also affected if it is used without an |
96 | explicit comparison function because it uses C<cmp> by default. |
97 | |
98 | B<Note:> C<eq> and C<ne> are unaffected by the locale: they always |
99 | perform a byte-by-byte comparison of their scalar operands. What's |
100 | more, if C<cmp> finds that its operands are equal according to the |
101 | collation sequence specified by the current locale, it goes on to |
102 | perform a byte-by-byte comparison, and only returns I<0> (equal) if the |
103 | operands are bit-for-bit identical. If you really want to know whether |
104 | two strings - which C<eq> and C<cmp> may consider different - are equal |
105 | as far as collation in the locale is concerned, see the discussion in |
106 | L<Category LC_COLLATE: Collation>. |
5f05dabc |
107 | |
108 | =item * |
109 | |
14280422 |
110 | B<Regular expressions and case-modification functions> (uc(), lc(), |
111 | ucfirst(), and lcfirst()) use C<LC_CTYPE> |
5f05dabc |
112 | |
113 | =item * |
114 | |
14280422 |
115 | B<The formatting functions> (printf(), sprintf() and write()) use |
5f05dabc |
116 | C<LC_NUMERIC> |
117 | |
118 | =item * |
119 | |
14280422 |
120 | B<The POSIX date formatting function> (strftime()) uses C<LC_TIME>. |
5f05dabc |
121 | |
122 | =back |
123 | |
14280422 |
124 | C<LC_COLLATE>, C<LC_CTYPE>, and so on, are discussed further in L<LOCALE |
125 | CATEGORIES>. |
5f05dabc |
126 | |
b0c42ed9 |
127 | The default behavior returns with S<C<no locale>> or on reaching the |
14280422 |
128 | end of the enclosing block. |
5f05dabc |
129 | |
14280422 |
130 | Note that the string result of any operation that uses locale |
131 | information is tainted, as it is possible for a locale to be |
132 | untrustworthy. See L<"SECURITY">. |
5f05dabc |
133 | |
134 | =head2 The setlocale function |
135 | |
14280422 |
136 | You can switch locales as often as you wish at run time with the |
137 | POSIX::setlocale() function: |
5f05dabc |
138 | |
139 | # This functionality not usable prior to Perl 5.004 |
140 | require 5.004; |
141 | |
142 | # Import locale-handling tool set from POSIX module. |
143 | # This example uses: setlocale -- the function call |
144 | # LC_CTYPE -- explained below |
145 | use POSIX qw(locale_h); |
146 | |
14280422 |
147 | # query and save the old locale |
5f05dabc |
148 | $old_locale = setlocale(LC_CTYPE); |
149 | |
150 | setlocale(LC_CTYPE, "fr_CA.ISO8859-1"); |
151 | # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1" |
152 | |
153 | setlocale(LC_CTYPE, ""); |
154 | # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG |
155 | # environment variables. See below for documentation. |
156 | |
157 | # restore the old locale |
158 | setlocale(LC_CTYPE, $old_locale); |
159 | |
14280422 |
160 | The first argument of setlocale() gives the B<category>, the second the |
161 | B<locale>. The category tells in what aspect of data processing you |
162 | want to apply locale-specific rules. Category names are discussed in |
163 | L<LOCALE CATEGORIES> and L<"ENVIRONMENT">. The locale is the name of a |
164 | collection of customization information corresponding to a particular |
165 | combination of language, country or territory, and codeset. Read on for |
166 | hints on the naming of locales: not all systems name locales as in the |
167 | example. |
168 | |
169 | If no second argument is provided, the function returns a string naming |
170 | the current locale for the category. You can use this value as the |
171 | second argument in a subsequent call to setlocale(). If a second |
5f05dabc |
172 | argument is given and it corresponds to a valid locale, the locale for |
173 | the category is set to that value, and the function returns the |
174 | now-current locale value. You can use this in a subsequent call to |
14280422 |
175 | setlocale(). (In some implementations, the return value may sometimes |
5f05dabc |
176 | differ from the value you gave as the second argument - think of it as |
177 | an alias for the value that you gave.) |
178 | |
179 | As the example shows, if the second argument is an empty string, the |
180 | category's locale is returned to the default specified by the |
181 | corresponding environment variables. Generally, this results in a |
182 | return to the default which was in force when Perl started up: changes |
14280422 |
183 | to the environment made by the application after start-up may or may not |
184 | be noticed, depending on the implementation of your system's C library. |
5f05dabc |
185 | |
14280422 |
186 | If the second argument does not correspond to a valid locale, the locale |
187 | for the category is not changed, and the function returns I<undef>. |
5f05dabc |
188 | |
14280422 |
189 | For further information about the categories, consult L<setlocale(3)>. |
190 | For the locales available in your system, also consult L<setlocale(3)> |
191 | and see whether it leads you to the list of the available locales |
192 | (search for the I<SEE ALSO> section). If that fails, try the following |
193 | command lines: |
5f05dabc |
194 | |
195 | locale -a |
196 | |
197 | nlsinfo |
198 | |
199 | ls /usr/lib/nls/loc |
200 | |
201 | ls /usr/lib/locale |
202 | |
203 | ls /usr/lib/nls |
204 | |
205 | and see whether they list something resembling these |
206 | |
2bdf8add |
207 | en_US.ISO8859-1 de_DE.ISO8859-1 ru_RU.ISO8859-5 |
208 | en_US de_DE ru_RU |
14280422 |
209 | en de ru |
2bdf8add |
210 | english german russian |
211 | english.iso88591 german.iso88591 russian.iso88595 |
5f05dabc |
212 | |
14280422 |
213 | Sadly, even though the calling interface for setlocale() has been |
2bdf8add |
214 | standardized, the names of the locales and the directories where |
215 | the configuration is, have not. The basic form of the name is |
216 | I<language_country/territory>B<.>I<codeset>, but the |
5f05dabc |
217 | latter parts are not always present. |
218 | |
14280422 |
219 | Two special locales are worth particular mention: "C" and "POSIX". |
220 | Currently these are effectively the same locale: the difference is |
221 | mainly that the first one is defined by the C standard and the second by |
222 | the POSIX standard. What they define is the B<default locale> in which |
223 | every program starts in the absence of locale information in its |
224 | environment. (The default default locale, if you will.) Its language |
225 | is (American) English and its character codeset ASCII. |
5f05dabc |
226 | |
14280422 |
227 | B<NOTE>: Not all systems have the "POSIX" locale (not all systems are |
228 | POSIX-conformant), so use "C" when you need explicitly to specify this |
229 | default locale. |
5f05dabc |
230 | |
231 | =head2 The localeconv function |
232 | |
14280422 |
233 | The POSIX::localeconv() function allows you to get particulars of the |
234 | locale-dependent numeric formatting information specified by the current |
235 | C<LC_NUMERIC> and C<LC_MONETARY> locales. (If you just want the name of |
236 | the current locale for a particular category, use POSIX::setlocale() |
237 | with a single parameter - see L<The setlocale function>.) |
5f05dabc |
238 | |
239 | use POSIX qw(locale_h); |
5f05dabc |
240 | |
241 | # Get a reference to a hash of locale-dependent info |
242 | $locale_values = localeconv(); |
243 | |
244 | # Output sorted list of the values |
245 | for (sort keys %$locale_values) { |
14280422 |
246 | printf "%-20s = %s\n", $_, $locale_values->{$_} |
5f05dabc |
247 | } |
248 | |
14280422 |
249 | localeconv() takes no arguments, and returns B<a reference to> a hash. |
250 | The keys of this hash are formatting variable names such as |
251 | C<decimal_point> and C<thousands_sep>; the values are the corresponding |
252 | values. See L<POSIX (3)/localeconv> for a longer example, which lists |
253 | all the categories an implementation might be expected to provide; some |
254 | provide more and others fewer, however. Note that you don't need C<use |
255 | locale>: as a function with the job of querying the locale, localeconv() |
256 | always observes the current locale. |
5f05dabc |
257 | |
258 | Here's a simple-minded example program which rewrites its command line |
259 | parameters as integers formatted correctly in the current locale: |
260 | |
261 | # See comments in previous example |
262 | require 5.004; |
263 | use POSIX qw(locale_h); |
5f05dabc |
264 | |
265 | # Get some of locale's numeric formatting parameters |
266 | my ($thousands_sep, $grouping) = |
14280422 |
267 | @{localeconv()}{'thousands_sep', 'grouping'}; |
5f05dabc |
268 | |
269 | # Apply defaults if values are missing |
270 | $thousands_sep = ',' unless $thousands_sep; |
271 | $grouping = 3 unless $grouping; |
272 | |
273 | # Format command line params for current locale |
14280422 |
274 | for (@ARGV) { |
275 | $_ = int; # Chop non-integer part |
5f05dabc |
276 | 1 while |
14280422 |
277 | s/(\d)(\d{$grouping}($|$thousands_sep))/$1$thousands_sep$2/; |
278 | print "$_"; |
5f05dabc |
279 | } |
280 | print "\n"; |
281 | |
5f05dabc |
282 | =head1 LOCALE CATEGORIES |
283 | |
14280422 |
284 | The subsections which follow describe basic locale categories. As well |
5f05dabc |
285 | as these, there are some combination categories which allow the |
14280422 |
286 | manipulation of more than one basic category at a time. See |
287 | L<"ENVIRONMENT"> for a discussion of these. |
5f05dabc |
288 | |
289 | =head2 Category LC_COLLATE: Collation |
290 | |
14280422 |
291 | When in the scope of S<C<use locale>>, Perl looks to the C<LC_COLLATE> |
5f05dabc |
292 | environment variable to determine the application's notions on the |
14280422 |
293 | collation (ordering) of characters. ('b' follows 'a' in Latin |
294 | alphabets, but where do 'E<aacute>' and 'E<aring>' belong?) |
5f05dabc |
295 | |
296 | Here is a code snippet that will tell you what are the alphanumeric |
297 | characters in the current locale, in the locale order: |
298 | |
299 | use locale; |
300 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
301 | |
14280422 |
302 | Compare this with the characters that you see and their order if you |
303 | state explicitly that the locale should be ignored: |
5f05dabc |
304 | |
305 | no locale; |
306 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
307 | |
308 | This machine-native collation (which is what you get unless S<C<use |
309 | locale>> has appeared earlier in the same block) must be used for |
310 | sorting raw binary data, whereas the locale-dependent collation of the |
b0c42ed9 |
311 | first example is useful for natural text. |
5f05dabc |
312 | |
14280422 |
313 | As noted in L<USING LOCALES>, C<cmp> compares according to the current |
314 | collation locale when C<use locale> is in effect, but falls back to a |
315 | byte-by-byte comparison for strings which the locale says are equal. You |
316 | can use POSIX::strcoll() if you don't want this fall-back: |
317 | |
318 | use POSIX qw(strcoll); |
319 | $equal_in_locale = |
320 | !strcoll("space and case ignored", "SpaceAndCaseIgnored"); |
321 | |
322 | $equal_in_locale will be true if the collation locale specifies a |
323 | dictionary-like ordering which ignores space characters completely, and |
324 | which folds case. Alternatively, you can use this idiom: |
325 | |
326 | use locale; |
327 | $s_a = "space and case ignored"; |
328 | $s_b = "SpaceAndCaseIgnored"; |
329 | $equal_in_locale = $s_a ge $s_b && $s_a le $s_b; |
330 | |
331 | which works because neither C<ne> nor C<ge> falls back to doing a |
332 | byte-by-byte comparison when the operands are equal according to the |
333 | locale. The idiom may be less efficient than using strcoll(), but, |
334 | unlike that function, it is not confused by strings containing embedded |
335 | nulls. |
336 | |
337 | If you have a single string which you want to check for "equality in |
338 | locale" against several others, you might think you could gain a little |
339 | efficiency by using POSIX::strxfrm() in conjunction with C<eq>: |
340 | |
341 | use POSIX qw(strxfrm); |
342 | $xfrm_string = strxfrm("Mixed-case string"); |
343 | print "locale collation ignores spaces\n" |
344 | if $xfrm_string eq strxfrm("Mixed-casestring"); |
345 | print "locale collation ignores hyphens\n" |
346 | if $xfrm_string eq strxfrm("Mixedcase string"); |
347 | print "locale collation ignores case\n" |
348 | if $xfrm_string eq strxfrm("mixed-case string"); |
349 | |
350 | strxfrm() takes a string and maps it into a transformed string for use |
351 | in byte-by-byte comparisons against other transformed strings during |
352 | collation. "Under the hood", locale-affected Perl comparison operators |
353 | call strxfrm() for both their operands, then do a byte-by-byte |
354 | comparison of the transformed strings. By calling strxfrm() explicitly, |
355 | and using a non locale-affected comparison, the example attempts to save |
356 | a couple of transformations. In fact, it doesn't save anything: Perl |
357 | magic (see L<perlguts/Magic>) creates the transformed version of a |
358 | string the first time it's needed in a comparison, then keeps it around |
359 | in case it's needed again. An example rewritten the easy way with |
360 | C<cmp> runs just about as fast. It also copes with null characters |
361 | embedded in strings; if you call strxfrm() directly, it treats the first |
362 | null it finds as a terminator. In short, don't call strxfrm() directly: |
363 | let Perl do it for you. |
364 | |
365 | Note: C<use locale> isn't shown in some of these examples, as it isn't |
366 | needed: strcoll() and strxfrm() exist only to generate locale-dependent |
367 | results, and so always obey the current C<LC_COLLATE> locale. |
5f05dabc |
368 | |
369 | =head2 Category LC_CTYPE: Character Types |
370 | |
371 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_CTYPE> locale |
14280422 |
372 | setting. This controls the application's notion of which characters are |
373 | alphabetic. This affects Perl's C<\w> regular expression metanotation, |
374 | which stands for alphanumeric characters - that is, alphabetic and |
375 | numeric characters. (Consult L<perlre> for more information about |
376 | regular expressions.) Thanks to C<LC_CTYPE>, depending on your locale |
377 | setting, characters like 'E<aelig>', 'E<eth>', 'E<szlig>', and |
378 | 'E<oslash>' may be understood as C<\w> characters. |
5f05dabc |
379 | |
380 | C<LC_CTYPE> also affects the POSIX character-class test functions - |
14280422 |
381 | isalpha(), islower() and so on. For example, if you move from the "C" |
382 | locale to a 7-bit Scandinavian one, you may find - possibly to your |
383 | surprise - that "|" moves from the ispunct() class to isalpha(). |
5f05dabc |
384 | |
14280422 |
385 | B<Note:> A broken or malicious C<LC_CTYPE> locale definition may result |
386 | in clearly ineligible characters being considered to be alphanumeric by |
387 | your application. For strict matching of (unaccented) letters and |
388 | digits - for example, in command strings - locale-aware applications |
389 | should use C<\w> inside a C<no locale> block. See L<"SECURITY">. |
5f05dabc |
390 | |
391 | =head2 Category LC_NUMERIC: Numeric Formatting |
392 | |
393 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_NUMERIC> |
14280422 |
394 | locale information, which controls application's idea of how numbers |
395 | should be formatted for human readability by the printf(), sprintf(), |
396 | and write() functions. String to numeric conversion by the |
397 | POSIX::strtod() function is also affected. In most implementations the |
398 | only effect is to change the character used for the decimal point - |
399 | perhaps from '.' to ',': these functions aren't aware of such niceties |
400 | as thousands separation and so on. (See L<The localeconv function> if |
401 | you care about these things.) |
402 | |
403 | Note that output produced by print() is B<never> affected by the |
5f05dabc |
404 | current locale: it is independent of whether C<use locale> or C<no |
14280422 |
405 | locale> is in effect, and corresponds to what you'd get from printf() |
5f05dabc |
406 | in the "C" locale. The same is true for Perl's internal conversions |
407 | between numeric and string formats: |
408 | |
409 | use POSIX qw(strtod); |
410 | use locale; |
14280422 |
411 | |
5f05dabc |
412 | $n = 5/2; # Assign numeric 2.5 to $n |
413 | |
414 | $a = " $n"; # Locale-independent conversion to string |
415 | |
416 | print "half five is $n\n"; # Locale-independent output |
417 | |
418 | printf "half five is %g\n", $n; # Locale-dependent output |
419 | |
14280422 |
420 | print "DECIMAL POINT IS COMMA\n" |
421 | if $n == (strtod("2,5"))[0]; # Locale-dependent conversion |
5f05dabc |
422 | |
423 | =head2 Category LC_MONETARY: Formatting of monetary amounts |
424 | |
14280422 |
425 | The C standard defines the C<LC_MONETARY> category, but no function that |
426 | is affected by its contents. (Those with experience of standards |
b0c42ed9 |
427 | committees will recognize that the working group decided to punt on the |
14280422 |
428 | issue.) Consequently, Perl takes no notice of it. If you really want |
429 | to use C<LC_MONETARY>, you can query its contents - see L<The localeconv |
430 | function> - and use the information that it returns in your |
b0c42ed9 |
431 | application's own formatting of currency amounts. However, you may well |
14280422 |
432 | find that the information, though voluminous and complex, does not quite |
433 | meet your requirements: currency formatting is a hard nut to crack. |
5f05dabc |
434 | |
435 | =head2 LC_TIME |
436 | |
14280422 |
437 | The output produced by POSIX::strftime(), which builds a formatted |
5f05dabc |
438 | human-readable date/time string, is affected by the current C<LC_TIME> |
439 | locale. Thus, in a French locale, the output produced by the C<%B> |
440 | format element (full month name) for the first month of the year would |
441 | be "janvier". Here's how to get a list of the long month names in the |
442 | current locale: |
443 | |
444 | use POSIX qw(strftime); |
14280422 |
445 | for (0..11) { |
446 | $long_month_name[$_] = |
447 | strftime("%B", 0, 0, 0, 1, $_, 96); |
5f05dabc |
448 | } |
449 | |
14280422 |
450 | Note: C<use locale> isn't needed in this example: as a function which |
451 | exists only to generate locale-dependent results, strftime() always |
452 | obeys the current C<LC_TIME> locale. |
5f05dabc |
453 | |
454 | =head2 Other categories |
455 | |
456 | The remaining locale category, C<LC_MESSAGES> (possibly supplemented by |
457 | others in particular implementations) is not currently used by Perl - |
b0c42ed9 |
458 | except possibly to affect the behavior of library functions called by |
14280422 |
459 | extensions which are not part of the standard Perl distribution. |
460 | |
461 | =head1 SECURITY |
462 | |
463 | While the main discussion of Perl security issues can be found in |
464 | L<perlsec>, a discussion of Perl's locale handling would be incomplete |
465 | if it did not draw your attention to locale-dependent security issues. |
466 | Locales - particularly on systems which allow unprivileged users to |
467 | build their own locales - are untrustworthy. A malicious (or just plain |
468 | broken) locale can make a locale-aware application give unexpected |
469 | results. Here are a few possibilities: |
470 | |
471 | =over 4 |
472 | |
473 | =item * |
474 | |
475 | Regular expression checks for safe file names or mail addresses using |
476 | C<\w> may be spoofed by an C<LC_CTYPE> locale which claims that |
477 | characters such as "E<gt>" and "|" are alphanumeric. |
478 | |
479 | =item * |
480 | |
481 | If the decimal point character in the C<LC_NUMERIC> locale is |
482 | surreptitiously changed from a dot to a comma, C<sprintf("%g", |
483 | 0.123456e3)> produces a string result of "123,456". Many people would |
484 | interpret this as one hundred and twenty-three thousand, four hundred |
485 | and fifty-six. |
486 | |
487 | =item * |
488 | |
489 | A sneaky C<LC_COLLATE> locale could result in the names of students with |
490 | "D" grades appearing ahead of those with "A"s. |
491 | |
492 | =item * |
493 | |
494 | An application which takes the trouble to use the information in |
495 | C<LC_MONETARY> may format debits as if they were credits and vice versa |
496 | if that locale has been subverted. Or it make may make payments in US |
497 | dollars instead of Hong Kong dollars. |
498 | |
499 | =item * |
500 | |
501 | The date and day names in dates formatted by strftime() could be |
502 | manipulated to advantage by a malicious user able to subvert the |
503 | C<LC_DATE> locale. ("Look - it says I wasn't in the building on |
504 | Sunday.") |
505 | |
506 | =back |
507 | |
508 | Such dangers are not peculiar to the locale system: any aspect of an |
509 | application's environment which may maliciously be modified presents |
510 | similar challenges. Similarly, they are not specific to Perl: any |
511 | programming language which allows you to write programs which take |
512 | account of their environment exposes you to these issues. |
513 | |
514 | Perl cannot protect you from all of the possibilities shown in the |
515 | examples - there is no substitute for your own vigilance - but, when |
516 | C<use locale> is in effect, Perl uses the tainting mechanism (see |
517 | L<perlsec>) to mark string results which become locale-dependent, and |
518 | which may be untrustworthy in consequence. Here is a summary of the |
b0c42ed9 |
519 | tainting behavior of operators and functions which may be affected by |
14280422 |
520 | the locale: |
521 | |
522 | =over 4 |
523 | |
524 | =item B<Comparison operators> (C<lt>, C<le>, C<ge>, C<gt> and C<cmp>): |
525 | |
526 | Scalar true/false (or less/equal/greater) result is never tainted. |
527 | |
528 | =item B<Matching operator> (C<m//>): |
529 | |
530 | Scalar true/false result never tainted. |
531 | |
532 | Subpatterns, either delivered as an array-context result, or as $1 etc. |
533 | are tainted if C<use locale> is in effect, and the subpattern regular |
534 | expression contains C<\w> (to match an alphanumeric character). The |
535 | matched pattern variable, $&, is also tainted if C<use locale> is in |
536 | effect, and the regular expression contains C<\w>. |
537 | |
538 | =item B<Substitution operator> (C<s///>): |
539 | |
b0c42ed9 |
540 | Has the same behavior as the match operator. When C<use locale> is |
14280422 |
541 | in effect, he left operand of C<=~> will become tainted if it is |
542 | modified as a result of a substitution based on a regular expression |
543 | match involving C<\w>. |
544 | |
545 | =item B<In-memory formatting function> (sprintf()): |
546 | |
547 | Result is tainted if "use locale" is in effect. |
548 | |
549 | =item B<Output formatting functions> (printf() and write()): |
550 | |
551 | Success/failure result is never tainted. |
552 | |
553 | =item B<Case-mapping functions> (lc(), lcfirst(), uc(), ucfirst()): |
554 | |
555 | Results are tainted if C<use locale> is in effect. |
556 | |
557 | =item B<POSIX locale-dependent functions> (localeconv(), strcoll(), |
558 | strftime(), strxfrm()): |
559 | |
560 | Results are never tainted. |
561 | |
562 | =item B<POSIX character class tests> (isalnum(), isalpha(), isdigit(), |
563 | isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(), |
564 | isxdigit()): |
565 | |
566 | True/false results are never tainted. |
567 | |
568 | =back |
569 | |
570 | Three examples illustrate locale-dependent tainting. |
571 | The first program, which ignores its locale, won't run: a value taken |
572 | directly from the command-line may not be used to name an output file |
573 | when taint checks are enabled. |
574 | |
575 | #/usr/local/bin/perl -T |
576 | # Run with taint checking |
577 | |
578 | # Command-line sanity check omitted... |
579 | $tainted_output_file = shift; |
580 | |
581 | open(F, ">$tainted_output_file") |
582 | or warn "Open of $untainted_output_file failed: $!\n"; |
583 | |
584 | The program can be made to run by "laundering" the tainted value through |
585 | a regular expression: the second example - which still ignores locale |
586 | information - runs, creating the file named on its command-line |
587 | if it can. |
588 | |
589 | #/usr/local/bin/perl -T |
590 | |
591 | $tainted_output_file = shift; |
592 | $tainted_output_file =~ m%[\w/]+%; |
593 | $untainted_output_file = $&; |
594 | |
595 | open(F, ">$untainted_output_file") |
596 | or warn "Open of $untainted_output_file failed: $!\n"; |
597 | |
598 | Compare this with a very similar program which is locale-aware: |
599 | |
600 | #/usr/local/bin/perl -T |
601 | |
602 | $tainted_output_file = shift; |
603 | use locale; |
604 | $tainted_output_file =~ m%[\w/]+%; |
605 | $localized_output_file = $&; |
606 | |
607 | open(F, ">$localized_output_file") |
608 | or warn "Open of $localized_output_file failed: $!\n"; |
609 | |
610 | This third program fails to run because $& is tainted: it is the result |
611 | of a match involving C<\w> when C<use locale> is in effect. |
5f05dabc |
612 | |
613 | =head1 ENVIRONMENT |
614 | |
615 | =over 12 |
616 | |
617 | =item PERL_BADLANG |
618 | |
14280422 |
619 | A string that can suppress Perl's warning about failed locale settings |
620 | at start-up. Failure can occur if the locale support in the operating |
621 | system is lacking (broken) is some way - or if you mistyped the name of |
622 | a locale when you set up your environment. If this environment variable |
623 | is absent, or has a value which does not evaluate to integer zero - that |
624 | is "0" or "" - Perl will complain about locale setting failures. |
5f05dabc |
625 | |
14280422 |
626 | B<NOTE>: PERL_BADLANG only gives you a way to hide the warning message. |
627 | The message tells about some problem in your system's locale support, |
628 | and you should investigate what the problem is. |
5f05dabc |
629 | |
630 | =back |
631 | |
632 | The following environment variables are not specific to Perl: They are |
14280422 |
633 | part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale() method |
634 | for controlling an application's opinion on data. |
5f05dabc |
635 | |
636 | =over 12 |
637 | |
638 | =item LC_ALL |
639 | |
640 | C<LC_ALL> is the "override-all" locale environment variable. If it is |
641 | set, it overrides all the rest of the locale environment variables. |
642 | |
643 | =item LC_CTYPE |
644 | |
645 | In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type |
646 | locale. In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG> |
647 | chooses the character type locale. |
648 | |
649 | =item LC_COLLATE |
650 | |
14280422 |
651 | In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation |
652 | (sorting) locale. In the absence of both C<LC_ALL> and C<LC_COLLATE>, |
653 | C<LANG> chooses the collation locale. |
5f05dabc |
654 | |
655 | =item LC_MONETARY |
656 | |
14280422 |
657 | In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the monetary |
658 | formatting locale. In the absence of both C<LC_ALL> and C<LC_MONETARY>, |
659 | C<LANG> chooses the monetary formatting locale. |
5f05dabc |
660 | |
661 | =item LC_NUMERIC |
662 | |
663 | In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format |
664 | locale. In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG> |
665 | chooses the numeric format. |
666 | |
667 | =item LC_TIME |
668 | |
14280422 |
669 | In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time |
670 | formatting locale. In the absence of both C<LC_ALL> and C<LC_TIME>, |
671 | C<LANG> chooses the date and time formatting locale. |
5f05dabc |
672 | |
673 | =item LANG |
674 | |
14280422 |
675 | C<LANG> is the "catch-all" locale environment variable. If it is set, it |
676 | is used as the last resort after the overall C<LC_ALL> and the |
5f05dabc |
677 | category-specific C<LC_...>. |
678 | |
679 | =back |
680 | |
681 | =head1 NOTES |
682 | |
683 | =head2 Backward compatibility |
684 | |
b0c42ed9 |
685 | Versions of Perl prior to 5.004 B<mostly> ignored locale information, |
686 | generally behaving as if something similar to the C<"C"> locale (see |
687 | L<The setlocale function>) was always in force, even if the program |
5f05dabc |
688 | environment suggested otherwise. By default, Perl still behaves this |
689 | way so as to maintain backward compatibility. If you want a Perl |
b0c42ed9 |
690 | application to pay attention to locale information, you B<must> use |
691 | the S<C<use locale>> pragma (see L<The S<C<use locale>> Pragma>) to |
692 | instruct it to do so. |
693 | |
694 | Versions of Perl from 5.002 to 5.003 did use the C<LC_CTYPE> |
695 | information if that was available, that is, C<\w> did understand what |
696 | are the letters according to the locale environment variables. |
697 | The problem was that the user had no control over the feature: |
698 | if the C library supported locales, Perl used them. |
699 | |
700 | =head2 I18N:Collate obsolete |
701 | |
702 | In versions of Perl prior to 5.004 per-locale collation was possible |
703 | using the C<I18N::Collate> library module. This module is now mildly |
704 | obsolete and should be avoided in new applications. The C<LC_COLLATE> |
705 | functionality is now integrated into the Perl core language: One can |
706 | use locale-specific scalar data completely normally with C<use locale>, |
707 | so there is no longer any need to juggle with the scalar references of |
708 | C<I18N::Collate>. |
5f05dabc |
709 | |
14280422 |
710 | =head2 Sort speed and memory use impacts |
5f05dabc |
711 | |
712 | Comparing and sorting by locale is usually slower than the default |
14280422 |
713 | sorting; slow-downs of two to four times have been observed. It will |
714 | also consume more memory: once a Perl scalar variable has participated |
715 | in any string comparison or sorting operation obeying the locale |
716 | collation rules, it will take 3-15 times more memory than before. (The |
717 | exact multiplier depends on the string's contents, the operating system |
718 | and the locale.) These downsides are dictated more by the operating |
719 | system's implementation of the locale system than by Perl. |
5f05dabc |
720 | |
5f05dabc |
721 | =head2 Freely available locale definitions |
722 | |
723 | There is a large collection of locale definitions at |
14280422 |
724 | C<ftp://dkuug.dk/i18n/WG15-collection>. You should be aware that it is |
725 | unsupported, and is not claimed to be fit for any purpose. If your |
726 | system allows the installation of arbitrary locales, you may find the |
727 | definitions useful as they are, or as a basis for the development of |
728 | your own locales. |
5f05dabc |
729 | |
14280422 |
730 | =head2 I18n and l10n |
5f05dabc |
731 | |
b0c42ed9 |
732 | "Internationalization" is often abbreviated as B<i18n> because its first |
733 | and last letters are separated by eighteen others. (You may guess why |
734 | the internalin ... internaliti ... i18n tends to get abbreviated.) In |
735 | the same way, "localization" is often abbreviated to B<l10n>. |
14280422 |
736 | |
737 | =head2 An imperfect standard |
738 | |
739 | Internationalization, as defined in the C and POSIX standards, can be |
740 | criticized as incomplete, ungainly, and having too large a granularity. |
741 | (Locales apply to a whole process, when it would arguably be more useful |
742 | to have them apply to a single thread, window group, or whatever.) They |
743 | also have a tendency, like standards groups, to divide the world into |
744 | nations, when we all know that the world can equally well be divided |
745 | into bankers, bikers, gamers, and so on. But, for now, it's the only |
746 | standard we've got. This may be construed as a bug. |
5f05dabc |
747 | |
748 | =head1 BUGS |
749 | |
750 | =head2 Broken systems |
751 | |
2bdf8add |
752 | In certain system environments the operating system's locale support |
753 | is broken and cannot be fixed or used by Perl. Such deficiencies can |
754 | and will result in mysterious hangs and/or Perl core dumps when the |
755 | C<use locale> is in effect. When confronted with such a system, |
756 | please report in excruciating detail to C<perlbug@perl.com>, and |
757 | complain to your vendor: maybe some bug fixes exist for these problems |
758 | in your operating system. Sometimes such bug fixes are called an |
759 | operating system upgrade. |
5f05dabc |
760 | |
761 | =head1 SEE ALSO |
762 | |
763 | L<POSIX (3)/isalnum>, L<POSIX (3)/isalpha>, L<POSIX (3)/isdigit>, |
764 | L<POSIX (3)/isgraph>, L<POSIX (3)/islower>, L<POSIX (3)/isprint>, |
765 | L<POSIX (3)/ispunct>, L<POSIX (3)/isspace>, L<POSIX (3)/isupper>, |
766 | L<POSIX (3)/isxdigit>, L<POSIX (3)/localeconv>, L<POSIX (3)/setlocale>, |
14280422 |
767 | L<POSIX (3)/strcoll>, L<POSIX (3)/strftime>, L<POSIX (3)/strtod>, |
768 | L<POSIX (3)/strxfrm> |
5f05dabc |
769 | |
770 | =head1 HISTORY |
771 | |
b0c42ed9 |
772 | Jarkko Hietaniemi's original F<perli18n.pod> heavily hacked by Dominic |
14280422 |
773 | Dunlop, assisted by the perl5-porters. |
5f05dabc |
774 | |
b0c42ed9 |
775 | Last update: Tue Dec 24 16:43:11 EST 1996 |