Update Changes.
[p5sagit/p5-mst-13.2.git] / pod / perlfaq6.pod
CommitLineData
68dc0745 1=head1 NAME
2
ae3d0b9f 3perlfaq6 - Regular Expressions ($Revision: 1.8 $, $Date: 2002/01/31 04:27:55 $)
68dc0745 4
5=head1 DESCRIPTION
6
7This section is surprisingly small because the rest of the FAQ is
8littered with answers involving regular expressions. For example,
9decoding a URL and checking whether something is a number are handled
10with regular expressions, but those answers are found elsewhere in
a6dd486b 11this document (in L<perlfaq9>: ``How do I decode or create those %-encodings
12on the web'' and L<perfaq4>: ``How do I determine whether a scalar is
13a number/whole/integer/float'', to be precise).
68dc0745 14
54310121 15=head2 How can I hope to use regular expressions without creating illegible and unmaintainable code?
68dc0745 16
17Three techniques can make regular expressions maintainable and
18understandable.
19
20=over 4
21
d92eb7b0 22=item Comments Outside the Regex
68dc0745 23
24Describe what you're doing and how you're doing it, using normal Perl
25comments.
26
27 # turn the line into the first word, a colon, and the
28 # number of characters on the rest of the line
5a964f20 29 s/^(\w+)(.*)/ lc($1) . ":" . length($2) /meg;
68dc0745 30
d92eb7b0 31=item Comments Inside the Regex
68dc0745 32
d92eb7b0 33The C</x> modifier causes whitespace to be ignored in a regex pattern
68dc0745 34(except in a character class), and also allows you to use normal
35comments there, too. As you can imagine, whitespace and comments help
36a lot.
37
38C</x> lets you turn this:
39
40 s{<(?:[^>'"]*|".*?"|'.*?')+>}{}gs;
41
42into this:
43
44 s{ < # opening angle bracket
45 (?: # Non-backreffing grouping paren
46 [^>'"] * # 0 or more things that are neither > nor ' nor "
47 | # or else
48 ".*?" # a section between double quotes (stingy match)
49 | # or else
50 '.*?' # a section between single quotes (stingy match)
51 ) + # all occurring one or more times
52 > # closing angle bracket
53 }{}gsx; # replace with nothing, i.e. delete
54
55It's still not quite so clear as prose, but it is very useful for
56describing the meaning of each part of the pattern.
57
58=item Different Delimiters
59
60While we normally think of patterns as being delimited with C</>
61characters, they can be delimited by almost any character. L<perlre>
62describes this. For example, the C<s///> above uses braces as
63delimiters. Selecting another delimiter can avoid quoting the
64delimiter within the pattern:
65
66 s/\/usr\/local/\/usr\/share/g; # bad delimiter choice
67 s#/usr/local#/usr/share#g; # better
68
69=back
70
71=head2 I'm having trouble matching over more than one line. What's wrong?
72
5a964f20 73Either you don't have more than one line in the string you're looking at
74(probably), or else you aren't using the correct modifier(s) on your
75pattern (possibly).
68dc0745 76
77There are many ways to get multiline data into a string. If you want
78it to happen automatically while reading input, you'll want to set $/
79(probably to '' for paragraphs or C<undef> for the whole file) to
80allow you to read more than one line at a time.
81
82Read L<perlre> to help you decide which of C</s> and C</m> (or both)
83you might want to use: C</s> allows dot to include newline, and C</m>
84allows caret and dollar to match next to a newline, not just at the
85end of the string. You do need to make sure that you've actually
86got a multiline string in there.
87
88For example, this program detects duplicate words, even when they span
89line breaks (but not paragraph ones). For this example, we don't need
90C</s> because we aren't using dot in a regular expression that we want
91to cross line boundaries. Neither do we need C</m> because we aren't
92wanting caret or dollar to match at any point inside the record next
93to newlines. But it's imperative that $/ be set to something other
94than the default, or else we won't actually ever have a multiline
95record read in.
96
97 $/ = ''; # read in more whole paragraph, not just one line
98 while ( <> ) {
5a964f20 99 while ( /\b([\w'-]+)(\s+\1)+\b/gi ) { # word starts alpha
68dc0745 100 print "Duplicate $1 at paragraph $.\n";
54310121 101 }
102 }
68dc0745 103
104Here's code that finds sentences that begin with "From " (which would
105be mangled by many mailers):
106
107 $/ = ''; # read in more whole paragraph, not just one line
108 while ( <> ) {
109 while ( /^From /gm ) { # /m makes ^ match next to \n
110 print "leading from in paragraph $.\n";
111 }
112 }
113
114Here's code that finds everything between START and END in a paragraph:
115
116 undef $/; # read in whole file, not just one line or paragraph
117 while ( <> ) {
fd89e497 118 while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries
68dc0745 119 print "$1\n";
120 }
121 }
122
123=head2 How can I pull out lines between two patterns that are themselves on different lines?
124
125You can use Perl's somewhat exotic C<..> operator (documented in
126L<perlop>):
127
128 perl -ne 'print if /START/ .. /END/' file1 file2 ...
129
130If you wanted text and not lines, you would use
131
65acb1b1 132 perl -0777 -ne 'print "$1\n" while /START(.*?)END/gs' file1 file2 ...
68dc0745 133
134But if you want nested occurrences of C<START> through C<END>, you'll
135run up against the problem described in the question in this section
136on matching balanced text.
137
5a964f20 138Here's another example of using C<..>:
139
140 while (<>) {
141 $in_header = 1 .. /^$/;
142 $in_body = /^$/ .. eof();
143 # now choose between them
144 } continue {
145 reset if eof(); # fix $.
146 }
147
68dc0745 148=head2 I put a regular expression into $/ but it didn't work. What's wrong?
149
150$/ must be a string, not a regular expression. Awk has to be better
151for something. :-)
152
fc36a67e 153Actually, you could do this if you don't mind reading the whole file
154into memory:
68dc0745 155
156 undef $/;
157 @records = split /your_pattern/, <FH>;
158
3fe9a6f1 159The Net::Telnet module (available from CPAN) has the capability to
160wait for a pattern in the input stream, or timeout if it doesn't
161appear within a certain time.
162
163 ## Create a file with three lines.
164 open FH, ">file";
165 print FH "The first line\nThe second line\nThe third line\n";
166 close FH;
167
168 ## Get a read/write filehandle to it.
169 $fh = new FileHandle "+<file";
170
171 ## Attach it to a "stream" object.
172 use Net::Telnet;
173 $file = new Net::Telnet (-fhopen => $fh);
174
175 ## Search for the second line and print out the third.
176 $file->waitfor('/second line\n/');
177 print $file->getline;
178
a6dd486b 179=head2 How do I substitute case insensitively on the LHS while preserving case on the RHS?
68dc0745 180
d92eb7b0 181Here's a lovely Perlish solution by Larry Rosler. It exploits
182properties of bitwise xor on ASCII strings.
183
184 $_= "this is a TEsT case";
185
186 $old = 'test';
187 $new = 'success';
188
575cc754 189 s{(\Q$old\E)}
d92eb7b0 190 { uc $new | (uc $1 ^ $1) .
191 (uc(substr $1, -1) ^ substr $1, -1) x
192 (length($new) - length $1)
193 }egi;
194
195 print;
196
8305e449 197And here it is as a subroutine, modeled after the above:
d92eb7b0 198
199 sub preserve_case($$) {
200 my ($old, $new) = @_;
201 my $mask = uc $old ^ $old;
202
203 uc $new | $mask .
204 substr($mask, -1) x (length($new) - length($old))
205 }
206
207 $a = "this is a TEsT case";
208 $a =~ s/(test)/preserve_case($1, "success")/egi;
209 print "$a\n";
210
211This prints:
212
213 this is a SUcCESS case
214
74b9445a 215As an alternative, to keep the case of the replacement word if it is
216longer than the original, you can use this code, by Jeff Pinyan:
217
218 sub preserve_case {
219 my ($from, $to) = @_;
220 my ($lf, $lt) = map length, @_;
7207e29d 221
74b9445a 222 if ($lt < $lf) { $from = substr $from, 0, $lt }
223 else { $from .= substr $to, $lf }
7207e29d 224
74b9445a 225 return uc $to | ($from ^ uc $from);
226 }
227
228This changes the sentence to "this is a SUcCess case."
229
d92eb7b0 230Just to show that C programmers can write C in any programming language,
231if you prefer a more C-like solution, the following script makes the
232substitution have the same case, letter by letter, as the original.
233(It also happens to run about 240% slower than the Perlish solution runs.)
234If the substitution has more characters than the string being substituted,
235the case of the last character is used for the rest of the substitution.
68dc0745 236
237 # Original by Nathan Torkington, massaged by Jeffrey Friedl
238 #
239 sub preserve_case($$)
240 {
241 my ($old, $new) = @_;
242 my ($state) = 0; # 0 = no change; 1 = lc; 2 = uc
243 my ($i, $oldlen, $newlen, $c) = (0, length($old), length($new));
244 my ($len) = $oldlen < $newlen ? $oldlen : $newlen;
245
246 for ($i = 0; $i < $len; $i++) {
247 if ($c = substr($old, $i, 1), $c =~ /[\W\d_]/) {
248 $state = 0;
249 } elsif (lc $c eq $c) {
250 substr($new, $i, 1) = lc(substr($new, $i, 1));
251 $state = 1;
252 } else {
253 substr($new, $i, 1) = uc(substr($new, $i, 1));
254 $state = 2;
255 }
256 }
257 # finish up with any remaining new (for when new is longer than old)
258 if ($newlen > $oldlen) {
259 if ($state == 1) {
260 substr($new, $oldlen) = lc(substr($new, $oldlen));
261 } elsif ($state == 2) {
262 substr($new, $oldlen) = uc(substr($new, $oldlen));
263 }
264 }
265 return $new;
266 }
267
5a964f20 268=head2 How can I make C<\w> match national character sets?
68dc0745 269
270See L<perllocale>.
271
272=head2 How can I match a locale-smart version of C</[a-zA-Z]/>?
273
274One alphabetic character would be C</[^\W\d_]/>, no matter what locale
54310121 275you're in. Non-alphabetics would be C</[\W\d_]/> (assuming you don't
68dc0745 276consider an underscore a letter).
277
d92eb7b0 278=head2 How can I quote a variable to use in a regex?
68dc0745 279
280The Perl parser will expand $variable and @variable references in
281regular expressions unless the delimiter is a single quote. Remember,
79a522f5 282too, that the right-hand side of a C<s///> substitution is considered
68dc0745 283a double-quoted string (see L<perlop> for more details). Remember
d92eb7b0 284also that any regex special characters will be acted on unless you
68dc0745 285precede the substitution with \Q. Here's an example:
286
287 $string = "to die?";
288 $lhs = "die?";
d92eb7b0 289 $rhs = "sleep, no more";
68dc0745 290
291 $string =~ s/\Q$lhs/$rhs/;
292 # $string is now "to sleep no more"
293
d92eb7b0 294Without the \Q, the regex would also spuriously match "di".
68dc0745 295
296=head2 What is C</o> really for?
297
46fc3d4c 298Using a variable in a regular expression match forces a re-evaluation
a6dd486b 299(and perhaps recompilation) each time the regular expression is
300encountered. The C</o> modifier locks in the regex the first time
301it's used. This always happens in a constant regular expression, and
302in fact, the pattern was compiled into the internal format at the same
303time your entire program was.
68dc0745 304
305Use of C</o> is irrelevant unless variable interpolation is used in
d92eb7b0 306the pattern, and if so, the regex engine will neither know nor care
68dc0745 307whether the variables change after the pattern is evaluated the I<very
308first> time.
309
310C</o> is often used to gain an extra measure of efficiency by not
311performing subsequent evaluations when you know it won't matter
312(because you know the variables won't change), or more rarely, when
d92eb7b0 313you don't want the regex to notice if they do.
68dc0745 314
315For example, here's a "paragrep" program:
316
317 $/ = ''; # paragraph mode
318 $pat = shift;
319 while (<>) {
320 print if /$pat/o;
321 }
322
323=head2 How do I use a regular expression to strip C style comments from a file?
324
325While this actually can be done, it's much harder than you'd think.
326For example, this one-liner
327
328 perl -0777 -pe 's{/\*.*?\*/}{}gs' foo.c
329
330will work in many but not all cases. You see, it's too simple-minded for
331certain kinds of C programs, in particular, those with what appear to be
332comments in quoted strings. For that, you'd need something like this,
d92eb7b0 333created by Jeffrey Friedl and later modified by Fred Curtis.
68dc0745 334
335 $/ = undef;
336 $_ = <>;
d92eb7b0 337 s#/\*[^*]*\*+([^/*][^*]*\*+)*/|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#$2#gs
68dc0745 338 print;
339
340This could, of course, be more legibly written with the C</x> modifier, adding
d92eb7b0 341whitespace and comments. Here it is expanded, courtesy of Fred Curtis.
342
343 s{
344 /\* ## Start of /* ... */ comment
345 [^*]*\*+ ## Non-* followed by 1-or-more *'s
346 (
347 [^/*][^*]*\*+
348 )* ## 0-or-more things which don't start with /
349 ## but do end with '*'
350 / ## End of /* ... */ comment
351
352 | ## OR various things which aren't comments:
353
354 (
355 " ## Start of " ... " string
356 (
357 \\. ## Escaped char
358 | ## OR
359 [^"\\] ## Non "\
360 )*
361 " ## End of " ... " string
362
363 | ## OR
364
365 ' ## Start of ' ... ' string
366 (
367 \\. ## Escaped char
368 | ## OR
369 [^'\\] ## Non '\
370 )*
371 ' ## End of ' ... ' string
372
373 | ## OR
374
375 . ## Anything other char
376 [^/"'\\]* ## Chars which doesn't start a comment, string or escape
377 )
378 }{$2}gxs;
379
380A slight modification also removes C++ comments:
381
382 s#/\*[^*]*\*+([^/*][^*]*\*+)*/|//[^\n]*|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#$2#gs;
68dc0745 383
384=head2 Can I use Perl regular expressions to match balanced text?
385
8305e449 386Historically, Perl regular expressions were not capable of matching
387balanced text. As of more recent versions of perl including 5.6.1
388experimental features have been added that make it possible to do this.
389Look at the documentation for the (??{ }) construct in recent perlre manual
390pages to see an example of matching balanced parentheses. Be sure to take
391special notice of the warnings present in the manual before making use
392of this feature.
393
394CPAN contains many modules that can be useful for matching text
395depending on the context. Damian Conway provides some useful
396patterns in Regexp::Common. The module Text::Balanced provides a
397general solution to this problem.
398
399One of the common applications of balanced text matching is working
400with XML and HTML. There are many modules available that support
401these needs. Two examples are HTML::Parser and XML::Parser. There
402are many others.
68dc0745 403
404An elaborate subroutine (for 7-bit ASCII only) to pull out balanced
405and possibly nested single chars, like C<`> and C<'>, C<{> and C<}>,
406or C<(> and C<)> can be found in
a93751fa 407http://www.cpan.org/authors/id/TOMC/scripts/pull_quotes.gz .
68dc0745 408
8305e449 409The C::Scan module from CPAN also contains such subs for internal use,
68dc0745 410but they are undocumented.
411
d92eb7b0 412=head2 What does it mean that regexes are greedy? How can I get around it?
68dc0745 413
d92eb7b0 414Most people mean that greedy regexes match as much as they can.
68dc0745 415Technically speaking, it's actually the quantifiers (C<?>, C<*>, C<+>,
416C<{}>) that are greedy rather than the whole pattern; Perl prefers local
417greed and immediate gratification to overall greed. To get non-greedy
418versions of the same quantifiers, use (C<??>, C<*?>, C<+?>, C<{}?>).
419
420An example:
421
422 $s1 = $s2 = "I am very very cold";
423 $s1 =~ s/ve.*y //; # I am cold
424 $s2 =~ s/ve.*?y //; # I am very cold
425
426Notice how the second substitution stopped matching as soon as it
427encountered "y ". The C<*?> quantifier effectively tells the regular
428expression engine to find a match as quickly as possible and pass
429control on to whatever is next in line, like you would if you were
430playing hot potato.
431
f9ac83b8 432=head2 How do I process each word on each line?
68dc0745 433
434Use the split function:
435
436 while (<>) {
fc36a67e 437 foreach $word ( split ) {
68dc0745 438 # do something with $word here
fc36a67e 439 }
54310121 440 }
68dc0745 441
54310121 442Note that this isn't really a word in the English sense; it's just
443chunks of consecutive non-whitespace characters.
68dc0745 444
f1cbbd6e 445To work with only alphanumeric sequences (including underscores), you
446might consider
68dc0745 447
448 while (<>) {
449 foreach $word (m/(\w+)/g) {
450 # do something with $word here
451 }
452 }
453
454=head2 How can I print out a word-frequency or line-frequency summary?
455
456To do this, you have to parse out each word in the input stream. We'll
54310121 457pretend that by word you mean chunk of alphabetics, hyphens, or
458apostrophes, rather than the non-whitespace chunk idea of a word given
68dc0745 459in the previous question:
460
461 while (<>) {
462 while ( /(\b[^\W_\d][\w'-]+\b)/g ) { # misses "`sheep'"
463 $seen{$1}++;
54310121 464 }
465 }
68dc0745 466 while ( ($word, $count) = each %seen ) {
467 print "$count $word\n";
54310121 468 }
68dc0745 469
470If you wanted to do the same thing for lines, you wouldn't need a
471regular expression:
472
fc36a67e 473 while (<>) {
68dc0745 474 $seen{$_}++;
54310121 475 }
68dc0745 476 while ( ($line, $count) = each %seen ) {
477 print "$count $line";
478 }
479
a6dd486b 480If you want these output in a sorted order, see L<perlfaq4>: ``How do I
481sort a hash (optionally by value instead of key)?''.
68dc0745 482
483=head2 How can I do approximate matching?
484
485See the module String::Approx available from CPAN.
486
487=head2 How do I efficiently match many regular expressions at once?
488
65acb1b1 489The following is extremely inefficient:
68dc0745 490
65acb1b1 491 # slow but obvious way
492 @popstates = qw(CO ON MI WI MN);
493 while (defined($line = <>)) {
494 for $state (@popstates) {
495 if ($line =~ /\b$state\b/i) {
496 print $line;
497 last;
498 }
499 }
500 }
501
502That's because Perl has to recompile all those patterns for each of
503the lines of the file. As of the 5.005 release, there's a much better
504approach, one which makes use of the new C<qr//> operator:
505
506 # use spiffy new qr// operator, with /i flag even
507 use 5.005;
508 @popstates = qw(CO ON MI WI MN);
509 @poppats = map { qr/\b$_\b/i } @popstates;
510 while (defined($line = <>)) {
511 for $patobj (@poppats) {
512 print $line if $line =~ /$patobj/;
513 }
68dc0745 514 }
515
516=head2 Why don't word-boundary searches with C<\b> work for me?
517
a6dd486b 518Two common misconceptions are that C<\b> is a synonym for C<\s+> and
68dc0745 519that it's the edge between whitespace characters and non-whitespace
520characters. Neither is correct. C<\b> is the place between a C<\w>
521character and a C<\W> character (that is, C<\b> is the edge of a
522"word"). It's a zero-width assertion, just like C<^>, C<$>, and all
523the other anchors, so it doesn't consume any characters. L<perlre>
d92eb7b0 524describes the behavior of all the regex metacharacters.
68dc0745 525
526Here are examples of the incorrect application of C<\b>, with fixes:
527
528 "two words" =~ /(\w+)\b(\w+)/; # WRONG
529 "two words" =~ /(\w+)\s+(\w+)/; # right
530
531 " =matchless= text" =~ /\b=(\w+)=\b/; # WRONG
532 " =matchless= text" =~ /=(\w+)=/; # right
533
534Although they may not do what you thought they did, C<\b> and C<\B>
535can still be quite useful. For an example of the correct use of
536C<\b>, see the example of matching duplicate words over multiple
537lines.
538
539An example of using C<\B> is the pattern C<\Bis\B>. This will find
540occurrences of "is" on the insides of words only, as in "thistle", but
541not "this" or "island".
542
543=head2 Why does using $&, $`, or $' slow my program down?
544
a6dd486b 545Once Perl sees that you need one of these variables anywhere in
546the program, it provides them on each and every pattern match.
65acb1b1 547The same mechanism that handles these provides for the use of $1, $2,
d92eb7b0 548etc., so you pay the same price for each regex that contains capturing
a6dd486b 549parentheses. If you never use $&, etc., in your script, then regexes
65acb1b1 550I<without> capturing parentheses won't be penalized. So avoid $&, $',
551and $` if you can, but if you can't, once you've used them at all, use
552them at will because you've already paid the price. Remember that some
553algorithms really appreciate them. As of the 5.005 release. the $&
554variable is no longer "expensive" the way the other two are.
68dc0745 555
556=head2 What good is C<\G> in a regular expression?
557
5d43e42d 558The notation C<\G> is used in a match or substitution in conjunction with
559the C</g> modifier to anchor the regular expression to the point just past
560where the last match occurred, i.e. the pos() point. A failed match resets
561the position of C<\G> unless the C</c> modifier is in effect. C<\G> can be
562used in a match without the C</g> modifier; it acts the same (i.e. still
563anchors at the pos() point) but of course only matches once and does not
564update pos(), as non-C</g> expressions never do. C<\G> in an expression
565applied to a target string that has never been matched against a C</g>
566expression before or has had its pos() reset is functionally equivalent to
567C<\A>, which matches at the beginning of the string.
68dc0745 568
569For example, suppose you had a line of text quoted in standard mail
c47ff5f1 570and Usenet notation, (that is, with leading C<< > >> characters), and
571you want change each leading C<< > >> into a corresponding C<:>. You
68dc0745 572could do so in this way:
573
574 s/^(>+)/':' x length($1)/gem;
575
576Or, using C<\G>, the much simpler (and faster):
577
578 s/\G>/:/g;
579
580A more sophisticated use might involve a tokenizer. The following
581lex-like example is courtesy of Jeffrey Friedl. It did not work in
c90c0ff4 5825.003 due to bugs in that release, but does work in 5.004 or better.
583(Note the use of C</c>, which prevents a failed match with C</g> from
584resetting the search position back to the beginning of the string.)
68dc0745 585
586 while (<>) {
587 chomp;
588 PARSER: {
c90c0ff4 589 m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; };
590 m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; };
591 m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; };
592 m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; };
68dc0745 593 }
594 }
595
596Of course, that could have been written as
597
598 while (<>) {
599 chomp;
600 PARSER: {
c90c0ff4 601 if ( /\G( \d+\b )/gcx {
68dc0745 602 print "number: $1\n";
603 redo PARSER;
604 }
c90c0ff4 605 if ( /\G( \w+ )/gcx {
68dc0745 606 print "word: $1\n";
607 redo PARSER;
608 }
c90c0ff4 609 if ( /\G( \s+ )/gcx {
68dc0745 610 print "space: $1\n";
611 redo PARSER;
612 }
c90c0ff4 613 if ( /\G( [^\w\d]+ )/gcx {
68dc0745 614 print "other: $1\n";
615 redo PARSER;
616 }
617 }
618 }
619
a6dd486b 620but then you lose the vertical alignment of the regular expressions.
68dc0745 621
d92eb7b0 622=head2 Are Perl regexes DFAs or NFAs? Are they POSIX compliant?
68dc0745 623
624While it's true that Perl's regular expressions resemble the DFAs
625(deterministic finite automata) of the egrep(1) program, they are in
46fc3d4c 626fact implemented as NFAs (non-deterministic finite automata) to allow
68dc0745 627backtracking and backreferencing. And they aren't POSIX-style either,
628because those guarantee worst-case behavior for all cases. (It seems
629that some people prefer guarantees of consistency, even when what's
630guaranteed is slowness.) See the book "Mastering Regular Expressions"
631(from O'Reilly) by Jeffrey Friedl for all the details you could ever
632hope to know on these matters (a full citation appears in
633L<perlfaq2>).
634
635=head2 What's wrong with using grep or map in a void context?
636
92c2ed05 637Both grep and map build a return list, regardless of their context.
638This means you're making Perl go to the trouble of building up a
639return list that you then just ignore. That's no way to treat a
640programming language, you insensitive scoundrel!
68dc0745 641
54310121 642=head2 How can I match strings with multibyte characters?
68dc0745 643
d9d154f2 644Starting from Perl 5.6 Perl has had some level of multibyte character
645support. Perl 5.8 or later is recommended. Supported multibyte
fe854a6f 646character repertoires include Unicode, and legacy encodings
d9d154f2 647through the Encode module. See L<perluniintro>, L<perlunicode>,
648and L<Encode>.
649
650If you are stuck with older Perls, you can do Unicode with the
651C<Unicode::String> module, and character conversions using the
652C<Unicode::Map8> and C<Unicode::Map> modules. If you are using
653Japanese encodings, you might try using the jperl 5.005_03.
654
655Finally, the following set of approaches was offered by Jeffrey
656Friedl, whose article in issue #5 of The Perl Journal talks about
657this very matter.
68dc0745 658
fc36a67e 659Let's suppose you have some weird Martian encoding where pairs of
660ASCII uppercase letters encode single Martian letters (i.e. the two
661bytes "CV" make a single Martian letter, as do the two bytes "SG",
662"VS", "XX", etc.). Other bytes represent single characters, just like
663ASCII.
68dc0745 664
fc36a67e 665So, the string of Martian "I am CVSGXX!" uses 12 bytes to encode the
666nine characters 'I', ' ', 'a', 'm', ' ', 'CV', 'SG', 'XX', '!'.
68dc0745 667
668Now, say you want to search for the single character C</GX/>. Perl
fc36a67e 669doesn't know about Martian, so it'll find the two bytes "GX" in the "I
670am CVSGXX!" string, even though that character isn't there: it just
671looks like it is because "SG" is next to "XX", but there's no real
672"GX". This is a big problem.
68dc0745 673
674Here are a few ways, all painful, to deal with it:
675
3fe9a6f1 676 $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian'' bytes
68dc0745 677 # are no longer adjacent.
678 print "found GX!\n" if $martian =~ /GX/;
679
680Or like this:
681
682 @chars = $martian =~ m/([A-Z][A-Z]|[^A-Z])/g;
683 # above is conceptually similar to: @chars = $text =~ m/(.)/g;
684 #
685 foreach $char (@chars) {
686 print "found GX!\n", last if $char eq 'GX';
687 }
688
689Or like this:
690
691 while ($martian =~ m/\G([A-Z][A-Z]|.)/gs) { # \G probably unneeded
54310121 692 print "found GX!\n", last if $1 eq 'GX';
68dc0745 693 }
694
695Or like this:
696
65acb1b1 697 die "sorry, Perl doesn't (yet) have Martian support )-:\n";
68dc0745 698
46fc3d4c 699There are many double- (and multi-) byte encodings commonly used these
68dc0745 700days. Some versions of these have 1-, 2-, 3-, and 4-byte characters,
701all mixed.
702
65acb1b1 703=head2 How do I match a pattern that is supplied by the user?
704
705Well, if it's really a pattern, then just use
706
707 chomp($pattern = <STDIN>);
708 if ($line =~ /$pattern/) { }
709
a6dd486b 710Alternatively, since you have no guarantee that your user entered
65acb1b1 711a valid regular expression, trap the exception this way:
712
713 if (eval { $line =~ /$pattern/ }) { }
714
a6dd486b 715If all you really want to search for a string, not a pattern,
65acb1b1 716then you should either use the index() function, which is made for
717string searching, or if you can't be disabused of using a pattern
718match on a non-pattern, then be sure to use C<\Q>...C<\E>, documented
719in L<perlre>.
720
721 $pattern = <STDIN>;
722
723 open (FILE, $input) or die "Couldn't open input $input: $!; aborting";
724 while (<FILE>) {
725 print if /\Q$pattern\E/;
726 }
727 close FILE;
728
68dc0745 729=head1 AUTHOR AND COPYRIGHT
730
0bc0ad85 731Copyright (c) 1997-2002 Tom Christiansen and Nathan Torkington.
5a964f20 732All rights reserved.
733
5a7beb56 734This documentation is free; you can redistribute it and/or modify it
735under the same terms as Perl itself.
5a964f20 736
737Irrespective of its distribution, all code examples in this file
738are hereby placed into the public domain. You are permitted and
739encouraged to use this code in your own programs for fun
740or for profit as you see fit. A simple comment in the code giving
741credit would be courteous but is not required.