Commit | Line | Data |
5f05dabc |
1 | =head1 NAME |
2 | |
b0c42ed9 |
3 | perllocale - Perl locale handling (internationalization and localization) |
5f05dabc |
4 | |
5 | =head1 DESCRIPTION |
6 | |
7 | Perl supports language-specific notions of data such as "is this a |
14280422 |
8 | letter", "what is the upper-case equivalent of this letter", and "which |
9 | of these letters comes first". These are important issues, especially |
10 | for languages other than English - but also for English: it would be |
11 | very naE<iuml>ve to think that C<A-Za-z> defines all the "letters". Perl |
12 | is also aware that some character other than '.' may be preferred as a |
13 | decimal point, and that output date representations may be |
14 | language-specific. The process of making an application take account of |
15 | its users' preferences in such matters is called B<internationalization> |
16 | (often abbreviated as B<i18n>); telling such an application about a |
17 | particular set of preferences is known as B<localization> (B<l10n>). |
18 | |
19 | Perl can understand language-specific data via the standardized (ISO C, |
20 | XPG4, POSIX 1.c) method called "the locale system". The locale system is |
b0c42ed9 |
21 | controlled per application using one pragma, one function call, and |
14280422 |
22 | several environment variables. |
23 | |
24 | B<NOTE>: This feature is new in Perl 5.004, and does not apply unless an |
25 | application specifically requests it - see L<Backward compatibility>. |
5f05dabc |
26 | |
27 | =head1 PREPARING TO USE LOCALES |
28 | |
14280422 |
29 | If Perl applications are to be able to understand and present your data |
30 | correctly according a locale of your choice, B<all> of the following |
5f05dabc |
31 | must be true: |
32 | |
33 | =over 4 |
34 | |
35 | =item * |
36 | |
37 | B<Your operating system must support the locale system>. If it does, |
14280422 |
38 | you should find that the setlocale() function is a documented part of |
5f05dabc |
39 | its C library. |
40 | |
41 | =item * |
42 | |
14280422 |
43 | B<Definitions for the locales which you use must be installed>. You, or |
44 | your system administrator, must make sure that this is the case. The |
45 | available locales, the location in which they are kept, and the manner |
46 | in which they are installed, vary from system to system. Some systems |
47 | provide only a few, hard-wired, locales, and do not allow more to be |
48 | added; others allow you to add "canned" locales provided by the system |
49 | supplier; still others allow you or the system administrator to define |
50 | and add arbitrary locales. (You may have to ask your supplier to |
51 | provide canned locales which are not delivered with your operating |
52 | system.) Read your system documentation for further illumination. |
5f05dabc |
53 | |
54 | =item * |
55 | |
56 | B<Perl must believe that the locale system is supported>. If it does, |
57 | C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is |
58 | C<define>. |
59 | |
60 | =back |
61 | |
62 | If you want a Perl application to process and present your data |
63 | according to a particular locale, the application code should include |
14280422 |
64 | the S<C<use locale>> pragma (see L<The use locale Pragma>) where |
5f05dabc |
65 | appropriate, and B<at least one> of the following must be true: |
66 | |
67 | =over 4 |
68 | |
69 | =item * |
70 | |
14280422 |
71 | B<The locale-determining environment variables (see L<"ENVIRONMENT">) |
72 | must be correctly set up>, either by yourself, or by the person who set |
73 | up your system account, at the time the application is started. |
5f05dabc |
74 | |
75 | =item * |
76 | |
14280422 |
77 | B<The application must set its own locale> using the method described in |
78 | L<The setlocale function>. |
5f05dabc |
79 | |
80 | =back |
81 | |
82 | =head1 USING LOCALES |
83 | |
84 | =head2 The use locale pragma |
85 | |
14280422 |
86 | By default, Perl ignores the current locale. The S<C<use locale>> |
87 | pragma tells Perl to use the current locale for some operations: |
5f05dabc |
88 | |
89 | =over 4 |
90 | |
91 | =item * |
92 | |
14280422 |
93 | B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>) and |
94 | the POSIX string collation functions strcoll() and strxfrm() use |
95 | C<LC_COLLATE>. sort() is also affected if it is used without an |
96 | explicit comparison function because it uses C<cmp> by default. |
97 | |
98 | B<Note:> C<eq> and C<ne> are unaffected by the locale: they always |
99 | perform a byte-by-byte comparison of their scalar operands. What's |
100 | more, if C<cmp> finds that its operands are equal according to the |
101 | collation sequence specified by the current locale, it goes on to |
102 | perform a byte-by-byte comparison, and only returns I<0> (equal) if the |
103 | operands are bit-for-bit identical. If you really want to know whether |
104 | two strings - which C<eq> and C<cmp> may consider different - are equal |
105 | as far as collation in the locale is concerned, see the discussion in |
106 | L<Category LC_COLLATE: Collation>. |
5f05dabc |
107 | |
108 | =item * |
109 | |
14280422 |
110 | B<Regular expressions and case-modification functions> (uc(), lc(), |
111 | ucfirst(), and lcfirst()) use C<LC_CTYPE> |
5f05dabc |
112 | |
113 | =item * |
114 | |
14280422 |
115 | B<The formatting functions> (printf(), sprintf() and write()) use |
5f05dabc |
116 | C<LC_NUMERIC> |
117 | |
118 | =item * |
119 | |
14280422 |
120 | B<The POSIX date formatting function> (strftime()) uses C<LC_TIME>. |
5f05dabc |
121 | |
122 | =back |
123 | |
14280422 |
124 | C<LC_COLLATE>, C<LC_CTYPE>, and so on, are discussed further in L<LOCALE |
125 | CATEGORIES>. |
5f05dabc |
126 | |
b0c42ed9 |
127 | The default behavior returns with S<C<no locale>> or on reaching the |
14280422 |
128 | end of the enclosing block. |
5f05dabc |
129 | |
14280422 |
130 | Note that the string result of any operation that uses locale |
131 | information is tainted, as it is possible for a locale to be |
132 | untrustworthy. See L<"SECURITY">. |
5f05dabc |
133 | |
134 | =head2 The setlocale function |
135 | |
14280422 |
136 | You can switch locales as often as you wish at run time with the |
137 | POSIX::setlocale() function: |
5f05dabc |
138 | |
139 | # This functionality not usable prior to Perl 5.004 |
140 | require 5.004; |
141 | |
142 | # Import locale-handling tool set from POSIX module. |
143 | # This example uses: setlocale -- the function call |
144 | # LC_CTYPE -- explained below |
145 | use POSIX qw(locale_h); |
146 | |
14280422 |
147 | # query and save the old locale |
5f05dabc |
148 | $old_locale = setlocale(LC_CTYPE); |
149 | |
150 | setlocale(LC_CTYPE, "fr_CA.ISO8859-1"); |
151 | # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1" |
152 | |
153 | setlocale(LC_CTYPE, ""); |
154 | # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG |
155 | # environment variables. See below for documentation. |
156 | |
157 | # restore the old locale |
158 | setlocale(LC_CTYPE, $old_locale); |
159 | |
14280422 |
160 | The first argument of setlocale() gives the B<category>, the second the |
161 | B<locale>. The category tells in what aspect of data processing you |
162 | want to apply locale-specific rules. Category names are discussed in |
163 | L<LOCALE CATEGORIES> and L<"ENVIRONMENT">. The locale is the name of a |
164 | collection of customization information corresponding to a particular |
165 | combination of language, country or territory, and codeset. Read on for |
166 | hints on the naming of locales: not all systems name locales as in the |
167 | example. |
168 | |
169 | If no second argument is provided, the function returns a string naming |
170 | the current locale for the category. You can use this value as the |
171 | second argument in a subsequent call to setlocale(). If a second |
5f05dabc |
172 | argument is given and it corresponds to a valid locale, the locale for |
173 | the category is set to that value, and the function returns the |
174 | now-current locale value. You can use this in a subsequent call to |
14280422 |
175 | setlocale(). (In some implementations, the return value may sometimes |
5f05dabc |
176 | differ from the value you gave as the second argument - think of it as |
177 | an alias for the value that you gave.) |
178 | |
179 | As the example shows, if the second argument is an empty string, the |
180 | category's locale is returned to the default specified by the |
181 | corresponding environment variables. Generally, this results in a |
182 | return to the default which was in force when Perl started up: changes |
14280422 |
183 | to the environment made by the application after start-up may or may not |
184 | be noticed, depending on the implementation of your system's C library. |
5f05dabc |
185 | |
14280422 |
186 | If the second argument does not correspond to a valid locale, the locale |
187 | for the category is not changed, and the function returns I<undef>. |
5f05dabc |
188 | |
14280422 |
189 | For further information about the categories, consult L<setlocale(3)>. |
190 | For the locales available in your system, also consult L<setlocale(3)> |
191 | and see whether it leads you to the list of the available locales |
192 | (search for the I<SEE ALSO> section). If that fails, try the following |
193 | command lines: |
5f05dabc |
194 | |
195 | locale -a |
196 | |
197 | nlsinfo |
198 | |
199 | ls /usr/lib/nls/loc |
200 | |
201 | ls /usr/lib/locale |
202 | |
203 | ls /usr/lib/nls |
204 | |
205 | and see whether they list something resembling these |
206 | |
14280422 |
207 | en de ru |
208 | english de_DE russian |
209 | english.iso88591 de_DE.ISO8859-1 russian.iso88595 |
210 | en_US german ru_RU |
211 | en_US.ISO8859-1 german.iso88591 ru_RU.ISO8859-5 |
5f05dabc |
212 | |
14280422 |
213 | Sadly, even though the calling interface for setlocale() has been |
5f05dabc |
214 | standardized, the names of the locales have not. The form of the name |
215 | is usually I<language_country>B</>I<territory>B<.>I<codeset>, but the |
216 | latter parts are not always present. |
217 | |
14280422 |
218 | Two special locales are worth particular mention: "C" and "POSIX". |
219 | Currently these are effectively the same locale: the difference is |
220 | mainly that the first one is defined by the C standard and the second by |
221 | the POSIX standard. What they define is the B<default locale> in which |
222 | every program starts in the absence of locale information in its |
223 | environment. (The default default locale, if you will.) Its language |
224 | is (American) English and its character codeset ASCII. |
5f05dabc |
225 | |
14280422 |
226 | B<NOTE>: Not all systems have the "POSIX" locale (not all systems are |
227 | POSIX-conformant), so use "C" when you need explicitly to specify this |
228 | default locale. |
5f05dabc |
229 | |
230 | =head2 The localeconv function |
231 | |
14280422 |
232 | The POSIX::localeconv() function allows you to get particulars of the |
233 | locale-dependent numeric formatting information specified by the current |
234 | C<LC_NUMERIC> and C<LC_MONETARY> locales. (If you just want the name of |
235 | the current locale for a particular category, use POSIX::setlocale() |
236 | with a single parameter - see L<The setlocale function>.) |
5f05dabc |
237 | |
238 | use POSIX qw(locale_h); |
5f05dabc |
239 | |
240 | # Get a reference to a hash of locale-dependent info |
241 | $locale_values = localeconv(); |
242 | |
243 | # Output sorted list of the values |
244 | for (sort keys %$locale_values) { |
14280422 |
245 | printf "%-20s = %s\n", $_, $locale_values->{$_} |
5f05dabc |
246 | } |
247 | |
14280422 |
248 | localeconv() takes no arguments, and returns B<a reference to> a hash. |
249 | The keys of this hash are formatting variable names such as |
250 | C<decimal_point> and C<thousands_sep>; the values are the corresponding |
251 | values. See L<POSIX (3)/localeconv> for a longer example, which lists |
252 | all the categories an implementation might be expected to provide; some |
253 | provide more and others fewer, however. Note that you don't need C<use |
254 | locale>: as a function with the job of querying the locale, localeconv() |
255 | always observes the current locale. |
5f05dabc |
256 | |
257 | Here's a simple-minded example program which rewrites its command line |
258 | parameters as integers formatted correctly in the current locale: |
259 | |
260 | # See comments in previous example |
261 | require 5.004; |
262 | use POSIX qw(locale_h); |
5f05dabc |
263 | |
264 | # Get some of locale's numeric formatting parameters |
265 | my ($thousands_sep, $grouping) = |
14280422 |
266 | @{localeconv()}{'thousands_sep', 'grouping'}; |
5f05dabc |
267 | |
268 | # Apply defaults if values are missing |
269 | $thousands_sep = ',' unless $thousands_sep; |
270 | $grouping = 3 unless $grouping; |
271 | |
272 | # Format command line params for current locale |
14280422 |
273 | for (@ARGV) { |
274 | $_ = int; # Chop non-integer part |
5f05dabc |
275 | 1 while |
14280422 |
276 | s/(\d)(\d{$grouping}($|$thousands_sep))/$1$thousands_sep$2/; |
277 | print "$_"; |
5f05dabc |
278 | } |
279 | print "\n"; |
280 | |
5f05dabc |
281 | =head1 LOCALE CATEGORIES |
282 | |
14280422 |
283 | The subsections which follow describe basic locale categories. As well |
5f05dabc |
284 | as these, there are some combination categories which allow the |
14280422 |
285 | manipulation of more than one basic category at a time. See |
286 | L<"ENVIRONMENT"> for a discussion of these. |
5f05dabc |
287 | |
288 | =head2 Category LC_COLLATE: Collation |
289 | |
14280422 |
290 | When in the scope of S<C<use locale>>, Perl looks to the C<LC_COLLATE> |
5f05dabc |
291 | environment variable to determine the application's notions on the |
14280422 |
292 | collation (ordering) of characters. ('b' follows 'a' in Latin |
293 | alphabets, but where do 'E<aacute>' and 'E<aring>' belong?) |
5f05dabc |
294 | |
295 | Here is a code snippet that will tell you what are the alphanumeric |
296 | characters in the current locale, in the locale order: |
297 | |
298 | use locale; |
299 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
300 | |
14280422 |
301 | Compare this with the characters that you see and their order if you |
302 | state explicitly that the locale should be ignored: |
5f05dabc |
303 | |
304 | no locale; |
305 | print +(sort grep /\w/, map { chr() } 0..255), "\n"; |
306 | |
307 | This machine-native collation (which is what you get unless S<C<use |
308 | locale>> has appeared earlier in the same block) must be used for |
309 | sorting raw binary data, whereas the locale-dependent collation of the |
b0c42ed9 |
310 | first example is useful for natural text. |
5f05dabc |
311 | |
14280422 |
312 | As noted in L<USING LOCALES>, C<cmp> compares according to the current |
313 | collation locale when C<use locale> is in effect, but falls back to a |
314 | byte-by-byte comparison for strings which the locale says are equal. You |
315 | can use POSIX::strcoll() if you don't want this fall-back: |
316 | |
317 | use POSIX qw(strcoll); |
318 | $equal_in_locale = |
319 | !strcoll("space and case ignored", "SpaceAndCaseIgnored"); |
320 | |
321 | $equal_in_locale will be true if the collation locale specifies a |
322 | dictionary-like ordering which ignores space characters completely, and |
323 | which folds case. Alternatively, you can use this idiom: |
324 | |
325 | use locale; |
326 | $s_a = "space and case ignored"; |
327 | $s_b = "SpaceAndCaseIgnored"; |
328 | $equal_in_locale = $s_a ge $s_b && $s_a le $s_b; |
329 | |
330 | which works because neither C<ne> nor C<ge> falls back to doing a |
331 | byte-by-byte comparison when the operands are equal according to the |
332 | locale. The idiom may be less efficient than using strcoll(), but, |
333 | unlike that function, it is not confused by strings containing embedded |
334 | nulls. |
335 | |
336 | If you have a single string which you want to check for "equality in |
337 | locale" against several others, you might think you could gain a little |
338 | efficiency by using POSIX::strxfrm() in conjunction with C<eq>: |
339 | |
340 | use POSIX qw(strxfrm); |
341 | $xfrm_string = strxfrm("Mixed-case string"); |
342 | print "locale collation ignores spaces\n" |
343 | if $xfrm_string eq strxfrm("Mixed-casestring"); |
344 | print "locale collation ignores hyphens\n" |
345 | if $xfrm_string eq strxfrm("Mixedcase string"); |
346 | print "locale collation ignores case\n" |
347 | if $xfrm_string eq strxfrm("mixed-case string"); |
348 | |
349 | strxfrm() takes a string and maps it into a transformed string for use |
350 | in byte-by-byte comparisons against other transformed strings during |
351 | collation. "Under the hood", locale-affected Perl comparison operators |
352 | call strxfrm() for both their operands, then do a byte-by-byte |
353 | comparison of the transformed strings. By calling strxfrm() explicitly, |
354 | and using a non locale-affected comparison, the example attempts to save |
355 | a couple of transformations. In fact, it doesn't save anything: Perl |
356 | magic (see L<perlguts/Magic>) creates the transformed version of a |
357 | string the first time it's needed in a comparison, then keeps it around |
358 | in case it's needed again. An example rewritten the easy way with |
359 | C<cmp> runs just about as fast. It also copes with null characters |
360 | embedded in strings; if you call strxfrm() directly, it treats the first |
361 | null it finds as a terminator. In short, don't call strxfrm() directly: |
362 | let Perl do it for you. |
363 | |
364 | Note: C<use locale> isn't shown in some of these examples, as it isn't |
365 | needed: strcoll() and strxfrm() exist only to generate locale-dependent |
366 | results, and so always obey the current C<LC_COLLATE> locale. |
5f05dabc |
367 | |
368 | =head2 Category LC_CTYPE: Character Types |
369 | |
370 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_CTYPE> locale |
14280422 |
371 | setting. This controls the application's notion of which characters are |
372 | alphabetic. This affects Perl's C<\w> regular expression metanotation, |
373 | which stands for alphanumeric characters - that is, alphabetic and |
374 | numeric characters. (Consult L<perlre> for more information about |
375 | regular expressions.) Thanks to C<LC_CTYPE>, depending on your locale |
376 | setting, characters like 'E<aelig>', 'E<eth>', 'E<szlig>', and |
377 | 'E<oslash>' may be understood as C<\w> characters. |
5f05dabc |
378 | |
379 | C<LC_CTYPE> also affects the POSIX character-class test functions - |
14280422 |
380 | isalpha(), islower() and so on. For example, if you move from the "C" |
381 | locale to a 7-bit Scandinavian one, you may find - possibly to your |
382 | surprise - that "|" moves from the ispunct() class to isalpha(). |
5f05dabc |
383 | |
14280422 |
384 | B<Note:> A broken or malicious C<LC_CTYPE> locale definition may result |
385 | in clearly ineligible characters being considered to be alphanumeric by |
386 | your application. For strict matching of (unaccented) letters and |
387 | digits - for example, in command strings - locale-aware applications |
388 | should use C<\w> inside a C<no locale> block. See L<"SECURITY">. |
5f05dabc |
389 | |
390 | =head2 Category LC_NUMERIC: Numeric Formatting |
391 | |
392 | When in the scope of S<C<use locale>>, Perl obeys the C<LC_NUMERIC> |
14280422 |
393 | locale information, which controls application's idea of how numbers |
394 | should be formatted for human readability by the printf(), sprintf(), |
395 | and write() functions. String to numeric conversion by the |
396 | POSIX::strtod() function is also affected. In most implementations the |
397 | only effect is to change the character used for the decimal point - |
398 | perhaps from '.' to ',': these functions aren't aware of such niceties |
399 | as thousands separation and so on. (See L<The localeconv function> if |
400 | you care about these things.) |
401 | |
402 | Note that output produced by print() is B<never> affected by the |
5f05dabc |
403 | current locale: it is independent of whether C<use locale> or C<no |
14280422 |
404 | locale> is in effect, and corresponds to what you'd get from printf() |
5f05dabc |
405 | in the "C" locale. The same is true for Perl's internal conversions |
406 | between numeric and string formats: |
407 | |
408 | use POSIX qw(strtod); |
409 | use locale; |
14280422 |
410 | |
5f05dabc |
411 | $n = 5/2; # Assign numeric 2.5 to $n |
412 | |
413 | $a = " $n"; # Locale-independent conversion to string |
414 | |
415 | print "half five is $n\n"; # Locale-independent output |
416 | |
417 | printf "half five is %g\n", $n; # Locale-dependent output |
418 | |
14280422 |
419 | print "DECIMAL POINT IS COMMA\n" |
420 | if $n == (strtod("2,5"))[0]; # Locale-dependent conversion |
5f05dabc |
421 | |
422 | =head2 Category LC_MONETARY: Formatting of monetary amounts |
423 | |
14280422 |
424 | The C standard defines the C<LC_MONETARY> category, but no function that |
425 | is affected by its contents. (Those with experience of standards |
b0c42ed9 |
426 | committees will recognize that the working group decided to punt on the |
14280422 |
427 | issue.) Consequently, Perl takes no notice of it. If you really want |
428 | to use C<LC_MONETARY>, you can query its contents - see L<The localeconv |
429 | function> - and use the information that it returns in your |
b0c42ed9 |
430 | application's own formatting of currency amounts. However, you may well |
14280422 |
431 | find that the information, though voluminous and complex, does not quite |
432 | meet your requirements: currency formatting is a hard nut to crack. |
5f05dabc |
433 | |
434 | =head2 LC_TIME |
435 | |
14280422 |
436 | The output produced by POSIX::strftime(), which builds a formatted |
5f05dabc |
437 | human-readable date/time string, is affected by the current C<LC_TIME> |
438 | locale. Thus, in a French locale, the output produced by the C<%B> |
439 | format element (full month name) for the first month of the year would |
440 | be "janvier". Here's how to get a list of the long month names in the |
441 | current locale: |
442 | |
443 | use POSIX qw(strftime); |
14280422 |
444 | for (0..11) { |
445 | $long_month_name[$_] = |
446 | strftime("%B", 0, 0, 0, 1, $_, 96); |
5f05dabc |
447 | } |
448 | |
14280422 |
449 | Note: C<use locale> isn't needed in this example: as a function which |
450 | exists only to generate locale-dependent results, strftime() always |
451 | obeys the current C<LC_TIME> locale. |
5f05dabc |
452 | |
453 | =head2 Other categories |
454 | |
455 | The remaining locale category, C<LC_MESSAGES> (possibly supplemented by |
456 | others in particular implementations) is not currently used by Perl - |
b0c42ed9 |
457 | except possibly to affect the behavior of library functions called by |
14280422 |
458 | extensions which are not part of the standard Perl distribution. |
459 | |
460 | =head1 SECURITY |
461 | |
462 | While the main discussion of Perl security issues can be found in |
463 | L<perlsec>, a discussion of Perl's locale handling would be incomplete |
464 | if it did not draw your attention to locale-dependent security issues. |
465 | Locales - particularly on systems which allow unprivileged users to |
466 | build their own locales - are untrustworthy. A malicious (or just plain |
467 | broken) locale can make a locale-aware application give unexpected |
468 | results. Here are a few possibilities: |
469 | |
470 | =over 4 |
471 | |
472 | =item * |
473 | |
474 | Regular expression checks for safe file names or mail addresses using |
475 | C<\w> may be spoofed by an C<LC_CTYPE> locale which claims that |
476 | characters such as "E<gt>" and "|" are alphanumeric. |
477 | |
478 | =item * |
479 | |
480 | If the decimal point character in the C<LC_NUMERIC> locale is |
481 | surreptitiously changed from a dot to a comma, C<sprintf("%g", |
482 | 0.123456e3)> produces a string result of "123,456". Many people would |
483 | interpret this as one hundred and twenty-three thousand, four hundred |
484 | and fifty-six. |
485 | |
486 | =item * |
487 | |
488 | A sneaky C<LC_COLLATE> locale could result in the names of students with |
489 | "D" grades appearing ahead of those with "A"s. |
490 | |
491 | =item * |
492 | |
493 | An application which takes the trouble to use the information in |
494 | C<LC_MONETARY> may format debits as if they were credits and vice versa |
495 | if that locale has been subverted. Or it make may make payments in US |
496 | dollars instead of Hong Kong dollars. |
497 | |
498 | =item * |
499 | |
500 | The date and day names in dates formatted by strftime() could be |
501 | manipulated to advantage by a malicious user able to subvert the |
502 | C<LC_DATE> locale. ("Look - it says I wasn't in the building on |
503 | Sunday.") |
504 | |
505 | =back |
506 | |
507 | Such dangers are not peculiar to the locale system: any aspect of an |
508 | application's environment which may maliciously be modified presents |
509 | similar challenges. Similarly, they are not specific to Perl: any |
510 | programming language which allows you to write programs which take |
511 | account of their environment exposes you to these issues. |
512 | |
513 | Perl cannot protect you from all of the possibilities shown in the |
514 | examples - there is no substitute for your own vigilance - but, when |
515 | C<use locale> is in effect, Perl uses the tainting mechanism (see |
516 | L<perlsec>) to mark string results which become locale-dependent, and |
517 | which may be untrustworthy in consequence. Here is a summary of the |
b0c42ed9 |
518 | tainting behavior of operators and functions which may be affected by |
14280422 |
519 | the locale: |
520 | |
521 | =over 4 |
522 | |
523 | =item B<Comparison operators> (C<lt>, C<le>, C<ge>, C<gt> and C<cmp>): |
524 | |
525 | Scalar true/false (or less/equal/greater) result is never tainted. |
526 | |
527 | =item B<Matching operator> (C<m//>): |
528 | |
529 | Scalar true/false result never tainted. |
530 | |
531 | Subpatterns, either delivered as an array-context result, or as $1 etc. |
532 | are tainted if C<use locale> is in effect, and the subpattern regular |
533 | expression contains C<\w> (to match an alphanumeric character). The |
534 | matched pattern variable, $&, is also tainted if C<use locale> is in |
535 | effect, and the regular expression contains C<\w>. |
536 | |
537 | =item B<Substitution operator> (C<s///>): |
538 | |
b0c42ed9 |
539 | Has the same behavior as the match operator. When C<use locale> is |
14280422 |
540 | in effect, he left operand of C<=~> will become tainted if it is |
541 | modified as a result of a substitution based on a regular expression |
542 | match involving C<\w>. |
543 | |
544 | =item B<In-memory formatting function> (sprintf()): |
545 | |
546 | Result is tainted if "use locale" is in effect. |
547 | |
548 | =item B<Output formatting functions> (printf() and write()): |
549 | |
550 | Success/failure result is never tainted. |
551 | |
552 | =item B<Case-mapping functions> (lc(), lcfirst(), uc(), ucfirst()): |
553 | |
554 | Results are tainted if C<use locale> is in effect. |
555 | |
556 | =item B<POSIX locale-dependent functions> (localeconv(), strcoll(), |
557 | strftime(), strxfrm()): |
558 | |
559 | Results are never tainted. |
560 | |
561 | =item B<POSIX character class tests> (isalnum(), isalpha(), isdigit(), |
562 | isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(), |
563 | isxdigit()): |
564 | |
565 | True/false results are never tainted. |
566 | |
567 | =back |
568 | |
569 | Three examples illustrate locale-dependent tainting. |
570 | The first program, which ignores its locale, won't run: a value taken |
571 | directly from the command-line may not be used to name an output file |
572 | when taint checks are enabled. |
573 | |
574 | #/usr/local/bin/perl -T |
575 | # Run with taint checking |
576 | |
577 | # Command-line sanity check omitted... |
578 | $tainted_output_file = shift; |
579 | |
580 | open(F, ">$tainted_output_file") |
581 | or warn "Open of $untainted_output_file failed: $!\n"; |
582 | |
583 | The program can be made to run by "laundering" the tainted value through |
584 | a regular expression: the second example - which still ignores locale |
585 | information - runs, creating the file named on its command-line |
586 | if it can. |
587 | |
588 | #/usr/local/bin/perl -T |
589 | |
590 | $tainted_output_file = shift; |
591 | $tainted_output_file =~ m%[\w/]+%; |
592 | $untainted_output_file = $&; |
593 | |
594 | open(F, ">$untainted_output_file") |
595 | or warn "Open of $untainted_output_file failed: $!\n"; |
596 | |
597 | Compare this with a very similar program which is locale-aware: |
598 | |
599 | #/usr/local/bin/perl -T |
600 | |
601 | $tainted_output_file = shift; |
602 | use locale; |
603 | $tainted_output_file =~ m%[\w/]+%; |
604 | $localized_output_file = $&; |
605 | |
606 | open(F, ">$localized_output_file") |
607 | or warn "Open of $localized_output_file failed: $!\n"; |
608 | |
609 | This third program fails to run because $& is tainted: it is the result |
610 | of a match involving C<\w> when C<use locale> is in effect. |
5f05dabc |
611 | |
612 | =head1 ENVIRONMENT |
613 | |
614 | =over 12 |
615 | |
616 | =item PERL_BADLANG |
617 | |
14280422 |
618 | A string that can suppress Perl's warning about failed locale settings |
619 | at start-up. Failure can occur if the locale support in the operating |
620 | system is lacking (broken) is some way - or if you mistyped the name of |
621 | a locale when you set up your environment. If this environment variable |
622 | is absent, or has a value which does not evaluate to integer zero - that |
623 | is "0" or "" - Perl will complain about locale setting failures. |
5f05dabc |
624 | |
14280422 |
625 | B<NOTE>: PERL_BADLANG only gives you a way to hide the warning message. |
626 | The message tells about some problem in your system's locale support, |
627 | and you should investigate what the problem is. |
5f05dabc |
628 | |
629 | =back |
630 | |
631 | The following environment variables are not specific to Perl: They are |
14280422 |
632 | part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale() method |
633 | for controlling an application's opinion on data. |
5f05dabc |
634 | |
635 | =over 12 |
636 | |
637 | =item LC_ALL |
638 | |
639 | C<LC_ALL> is the "override-all" locale environment variable. If it is |
640 | set, it overrides all the rest of the locale environment variables. |
641 | |
642 | =item LC_CTYPE |
643 | |
644 | In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type |
645 | locale. In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG> |
646 | chooses the character type locale. |
647 | |
648 | =item LC_COLLATE |
649 | |
14280422 |
650 | In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation |
651 | (sorting) locale. In the absence of both C<LC_ALL> and C<LC_COLLATE>, |
652 | C<LANG> chooses the collation locale. |
5f05dabc |
653 | |
654 | =item LC_MONETARY |
655 | |
14280422 |
656 | In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the monetary |
657 | formatting locale. In the absence of both C<LC_ALL> and C<LC_MONETARY>, |
658 | C<LANG> chooses the monetary formatting locale. |
5f05dabc |
659 | |
660 | =item LC_NUMERIC |
661 | |
662 | In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format |
663 | locale. In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG> |
664 | chooses the numeric format. |
665 | |
666 | =item LC_TIME |
667 | |
14280422 |
668 | In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time |
669 | formatting locale. In the absence of both C<LC_ALL> and C<LC_TIME>, |
670 | C<LANG> chooses the date and time formatting locale. |
5f05dabc |
671 | |
672 | =item LANG |
673 | |
14280422 |
674 | C<LANG> is the "catch-all" locale environment variable. If it is set, it |
675 | is used as the last resort after the overall C<LC_ALL> and the |
5f05dabc |
676 | category-specific C<LC_...>. |
677 | |
678 | =back |
679 | |
680 | =head1 NOTES |
681 | |
682 | =head2 Backward compatibility |
683 | |
b0c42ed9 |
684 | Versions of Perl prior to 5.004 B<mostly> ignored locale information, |
685 | generally behaving as if something similar to the C<"C"> locale (see |
686 | L<The setlocale function>) was always in force, even if the program |
5f05dabc |
687 | environment suggested otherwise. By default, Perl still behaves this |
688 | way so as to maintain backward compatibility. If you want a Perl |
b0c42ed9 |
689 | application to pay attention to locale information, you B<must> use |
690 | the S<C<use locale>> pragma (see L<The S<C<use locale>> Pragma>) to |
691 | instruct it to do so. |
692 | |
693 | Versions of Perl from 5.002 to 5.003 did use the C<LC_CTYPE> |
694 | information if that was available, that is, C<\w> did understand what |
695 | are the letters according to the locale environment variables. |
696 | The problem was that the user had no control over the feature: |
697 | if the C library supported locales, Perl used them. |
698 | |
699 | =head2 I18N:Collate obsolete |
700 | |
701 | In versions of Perl prior to 5.004 per-locale collation was possible |
702 | using the C<I18N::Collate> library module. This module is now mildly |
703 | obsolete and should be avoided in new applications. The C<LC_COLLATE> |
704 | functionality is now integrated into the Perl core language: One can |
705 | use locale-specific scalar data completely normally with C<use locale>, |
706 | so there is no longer any need to juggle with the scalar references of |
707 | C<I18N::Collate>. |
5f05dabc |
708 | |
14280422 |
709 | =head2 Sort speed and memory use impacts |
5f05dabc |
710 | |
711 | Comparing and sorting by locale is usually slower than the default |
14280422 |
712 | sorting; slow-downs of two to four times have been observed. It will |
713 | also consume more memory: once a Perl scalar variable has participated |
714 | in any string comparison or sorting operation obeying the locale |
715 | collation rules, it will take 3-15 times more memory than before. (The |
716 | exact multiplier depends on the string's contents, the operating system |
717 | and the locale.) These downsides are dictated more by the operating |
718 | system's implementation of the locale system than by Perl. |
5f05dabc |
719 | |
5f05dabc |
720 | =head2 Freely available locale definitions |
721 | |
722 | There is a large collection of locale definitions at |
14280422 |
723 | C<ftp://dkuug.dk/i18n/WG15-collection>. You should be aware that it is |
724 | unsupported, and is not claimed to be fit for any purpose. If your |
725 | system allows the installation of arbitrary locales, you may find the |
726 | definitions useful as they are, or as a basis for the development of |
727 | your own locales. |
5f05dabc |
728 | |
14280422 |
729 | =head2 I18n and l10n |
5f05dabc |
730 | |
b0c42ed9 |
731 | "Internationalization" is often abbreviated as B<i18n> because its first |
732 | and last letters are separated by eighteen others. (You may guess why |
733 | the internalin ... internaliti ... i18n tends to get abbreviated.) In |
734 | the same way, "localization" is often abbreviated to B<l10n>. |
14280422 |
735 | |
736 | =head2 An imperfect standard |
737 | |
738 | Internationalization, as defined in the C and POSIX standards, can be |
739 | criticized as incomplete, ungainly, and having too large a granularity. |
740 | (Locales apply to a whole process, when it would arguably be more useful |
741 | to have them apply to a single thread, window group, or whatever.) They |
742 | also have a tendency, like standards groups, to divide the world into |
743 | nations, when we all know that the world can equally well be divided |
744 | into bankers, bikers, gamers, and so on. But, for now, it's the only |
745 | standard we've got. This may be construed as a bug. |
5f05dabc |
746 | |
747 | =head1 BUGS |
748 | |
749 | =head2 Broken systems |
750 | |
14280422 |
751 | In certain system environments the operating system's locale support is |
752 | broken and cannot be fixed or used by Perl. Such deficiencies can and |
753 | will result in mysterious hangs and/or Perl core dumps. One example is |
754 | IRIX before release 6.2, in which the C<LC_COLLATE> support simply does |
755 | not work. When confronted with such a system, please report in |
756 | excruciating detail to C<perlbug@perl.com>, and complain to your vendor: |
757 | maybe some bug fixes exist for these problems in your operating system. |
758 | Sometimes such bug fixes are called an operating system upgrade. |
5f05dabc |
759 | |
760 | =head1 SEE ALSO |
761 | |
762 | L<POSIX (3)/isalnum>, L<POSIX (3)/isalpha>, L<POSIX (3)/isdigit>, |
763 | L<POSIX (3)/isgraph>, L<POSIX (3)/islower>, L<POSIX (3)/isprint>, |
764 | L<POSIX (3)/ispunct>, L<POSIX (3)/isspace>, L<POSIX (3)/isupper>, |
765 | L<POSIX (3)/isxdigit>, L<POSIX (3)/localeconv>, L<POSIX (3)/setlocale>, |
14280422 |
766 | L<POSIX (3)/strcoll>, L<POSIX (3)/strftime>, L<POSIX (3)/strtod>, |
767 | L<POSIX (3)/strxfrm> |
5f05dabc |
768 | |
769 | =head1 HISTORY |
770 | |
b0c42ed9 |
771 | Jarkko Hietaniemi's original F<perli18n.pod> heavily hacked by Dominic |
14280422 |
772 | Dunlop, assisted by the perl5-porters. |
5f05dabc |
773 | |
b0c42ed9 |
774 | Last update: Tue Dec 24 16:43:11 EST 1996 |