X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlfaq6.pod;h=ed0bc297412e18f35335cde2fae99a42d7c642c6;hb=f26f4a2f8b63c72a33468ddeeb9d0337f0892af6;hp=4e5ba50ee5167b880fa638aaaa732d1b3898c98e;hpb=3392b9ec92d57049a2d2a6e9d3cae41fff9ee2de;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlfaq6.pod b/pod/perlfaq6.pod index 4e5ba50..ed0bc29 100644 --- a/pod/perlfaq6.pod +++ b/pod/perlfaq6.pod @@ -1,6 +1,6 @@ =head1 NAME -perlfaq6 - Regular Expressions ($Revision: 1.12 $, $Date: 2002/06/01 22:31:09 $) +perlfaq6 - Regular Expressions ($Revision: 1.35 $, $Date: 2005/08/10 15:55:08 $) =head1 DESCRIPTION @@ -8,9 +8,9 @@ This section is surprisingly small because the rest of the FAQ is littered with answers involving regular expressions. For example, decoding a URL and checking whether something is a number are handled with regular expressions, but those answers are found elsewhere in -this document (in L: ``How do I decode or create those %-encodings -on the web'' and L: ``How do I determine whether a scalar is -a number/whole/integer/float'', to be precise). +this document (in L: "How do I decode or create those %-encodings +on the web" and L: "How do I determine whether a scalar is +a number/whole/integer/float", to be precise). =head2 How can I hope to use regular expressions without creating illegible and unmaintainable code? @@ -143,38 +143,50 @@ Here's another example of using C<..>: # now choose between them } continue { reset if eof(); # fix $. - } + } =head2 I put a regular expression into $/ but it didn't work. What's wrong? -$/ must be a string, not a regular expression. Awk has to be better -for something. :-) +Up to Perl 5.8.0, $/ has to be a string. This may change in 5.10, +but don't get your hopes up. Until then, you can use these examples +if you really need to do this. + +If you have File::Stream, this is easy. + + use File::Stream; + my $stream = File::Stream->new( + $filehandle, + separator => qr/\s*,\s*/, + ); -Actually, you could do this if you don't mind reading the whole file -into memory: + print "$_\n" while <$stream>; - undef $/; - @records = split /your_pattern/, ; +If you don't have File::Stream, you have to do a little more work. -The Net::Telnet module (available from CPAN) has the capability to -wait for a pattern in the input stream, or timeout if it doesn't -appear within a certain time. +You can use the four argument form of sysread to continually add to +a buffer. After you add to the buffer, you check if you have a +complete line (using your regular expression). - ## Create a file with three lines. - open FH, ">file"; - print FH "The first line\nThe second line\nThe third line\n"; - close FH; + local $_ = ""; + while( sysread FH, $_, 8192, length ) { + while( s/^((?s).*?)your_pattern/ ) { + my $record = $1; + # do stuff here. + } + } - ## Get a read/write filehandle to it. - $fh = new IO::File "+ $fh); + local $_ = ""; + while( sysread FH, $_, 8192, length ) { + foreach my $record ( m/\G((?s).*?)your_pattern/gc ) { + # do stuff here. + } + substr( $_, 0, pos ) = "" if pos; + } - ## Search for the second line and print out the third. - $file->waitfor('/second line\n/'); - print $file->getline; =head2 How do I substitute case insensitively on the LHS while preserving case on the RHS? @@ -201,7 +213,7 @@ And here it is as a subroutine, modeled after the above: my $mask = uc $old ^ $old; uc $new | $mask . - substr($mask, -1) x (length($new) - length($old)) + substr($mask, -1) x (length($new) - length($old)) } $a = "this is a TEsT case"; @@ -267,13 +279,21 @@ the case of the last character is used for the rest of the substitution. =head2 How can I make C<\w> match national character sets? -See L. +Put C in your script. The \w character class is taken +from the current locale. + +See L for details. =head2 How can I match a locale-smart version of C? -One alphabetic character would be C, no matter what locale -you're in. Non-alphabetics would be C (assuming you don't -consider an underscore a letter). +You can use the POSIX character class syntax C +documented in L. + +No matter which locale you are in, the alphabetic characters are +the characters in \w without the digits and the underscore. +As a regex, that looks like C. Its complement, +the non-alphabetics, is then everything in \W along with +the digits and the underscore, or C. =head2 How can I quote a variable to use in a regex? @@ -284,14 +304,26 @@ a double-quoted string (see L for more details). Remember also that any regex special characters will be acted on unless you precede the substitution with \Q. Here's an example: - $string = "to die?"; - $lhs = "die?"; - $rhs = "sleep, no more"; + $string = "Placido P. Octopus"; + $regex = "P."; + + $string =~ s/$regex/Polyp/; + # $string is now "Polypacido P. Octopus" + +Because C<.> is special in regular expressions, and can match any +single character, the regex C here has matched the in the +original string. + +To escape the special meaning of C<.>, we use C<\Q>: + + $string = "Placido P. Octopus"; + $regex = "P."; - $string =~ s/\Q$lhs/$rhs/; - # $string is now "to sleep no more" + $string =~ s/\Q$regex/Polyp/; + # $string is now "Placido Polyp Octopus" -Without the \Q, the regex would also spuriously match "di". +The use of C<\Q> causes the <.> in the regex to be treated as a +regular character, so that C matches a C

followed by a dot. =head2 What is C really for? @@ -334,7 +366,7 @@ created by Jeffrey Friedl and later modified by Fred Curtis. $/ = undef; $_ = <>; - s#/\*[^*]*\*+([^/*][^*]*\*+)*/|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#$2#gs + s#/\*[^*]*\*+([^/*][^*]*\*+)*/|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $2 ? $2 : ""#gse; print; This could, of course, be more legibly written with the C modifier, adding @@ -375,11 +407,11 @@ whitespace and comments. Here it is expanded, courtesy of Fred Curtis. . ## Anything other char [^/"'\\]* ## Chars which doesn't start a comment, string or escape ) - }{$2}gxs; + }{defined $2 ? $2 : ""}gxse; A slight modification also removes C++ comments: - s#/\*[^*]*\*+([^/*][^*]*\*+)*/|//[^\n]*|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#$2#gs; + s#/\*[^*]*\*+([^/*][^*]*\*+)*/|//[^\n]*|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $2 ? $2 : ""#gse; =head2 Can I use Perl regular expressions to match balanced text? @@ -434,9 +466,9 @@ playing hot potato. Use the split function: while (<>) { - foreach $word ( split ) { + foreach $word ( split ) { # do something with $word here - } + } } Note that this isn't really a word in the English sense; it's just @@ -470,15 +502,15 @@ in the previous question: If you wanted to do the same thing for lines, you wouldn't need a regular expression: - while (<>) { + while (<>) { $seen{$_}++; } while ( ($line, $count) = each %seen ) { print "$count $line"; } -If you want these output in a sorted order, see L: ``How do I -sort a hash (optionally by value instead of key)?''. +If you want these output in a sorted order, see L: "How do I +sort a hash (optionally by value instead of key)?". =head2 How can I do approximate matching? @@ -486,138 +518,228 @@ See the module String::Approx available from CPAN. =head2 How do I efficiently match many regular expressions at once? -The following is extremely inefficient: - - # slow but obvious way - @popstates = qw(CO ON MI WI MN); - while (defined($line = <>)) { - for $state (@popstates) { - if ($line =~ /\b$state\b/i) { - print $line; - last; - } - } - } - -That's because Perl has to recompile all those patterns for each of -the lines of the file. As of the 5.005 release, there's a much better -approach, one which makes use of the new C operator: - - # use spiffy new qr// operator, with /i flag even - use 5.005; - @popstates = qw(CO ON MI WI MN); - @poppats = map { qr/\b$_\b/i } @popstates; - while (defined($line = <>)) { - for $patobj (@poppats) { - print $line if $line =~ /$patobj/; - } - } +( contributed by brian d foy ) + +Avoid asking Perl to compile a regular expression every time +you want to match it. In this example, perl must recompile +the regular expression for every iteration of the foreach() +loop since it has no way to know what $pattern will be. + + @patterns = qw( foo bar baz ); + + LINE: while( <> ) + { + foreach $pattern ( @patterns ) + { + print if /\b$pattern\b/i; + next LINE; + } + } + +The qr// operator showed up in perl 5.005. It compiles a +regular expression, but doesn't apply it. When you use the +pre-compiled version of the regex, perl does less work. In +this example, I inserted a map() to turn each pattern into +its pre-compiled form. The rest of the script is the same, +but faster. + + @patterns = map { qr/\b$_\b/i } qw( foo bar baz ); + + LINE: while( <> ) + { + foreach $pattern ( @patterns ) + { + print if /\b$pattern\b/i; + next LINE; + } + } + +In some cases, you may be able to make several patterns into +a single regular expression. Beware of situations that require +backtracking though. + + $regex = join '|', qw( foo bar baz ); + + LINE: while( <> ) + { + print if /\b(?:$regex)\b/i; + } + +For more details on regular expression efficiency, see Mastering +Regular Expressions by Jeffrey Freidl. He explains how regular +expressions engine work and why some patterns are surprisingly +inefficient. Once you understand how perl applies regular +expressions, you can tune them for individual situations. =head2 Why don't word-boundary searches with C<\b> work for me? -Two common misconceptions are that C<\b> is a synonym for C<\s+> and -that it's the edge between whitespace characters and non-whitespace -characters. Neither is correct. C<\b> is the place between a C<\w> -character and a C<\W> character (that is, C<\b> is the edge of a -"word"). It's a zero-width assertion, just like C<^>, C<$>, and all -the other anchors, so it doesn't consume any characters. L -describes the behavior of all the regex metacharacters. +(contributed by brian d foy) -Here are examples of the incorrect application of C<\b>, with fixes: +Ensure that you know what \b really does: it's the boundary between a +word character, \w, and something that isn't a word character. That +thing that isn't a word character might be \W, but it can also be the +start or end of the string. - "two words" =~ /(\w+)\b(\w+)/; # WRONG - "two words" =~ /(\w+)\s+(\w+)/; # right +It's not (not!) the boundary between whitespace and non-whitespace, +and it's not the stuff between words we use to create sentences. - " =matchless= text" =~ /\b=(\w+)=\b/; # WRONG - " =matchless= text" =~ /=(\w+)=/; # right +In regex speak, a word boundary (\b) is a "zero width assertion", +meaning that it doesn't represent a character in the string, but a +condition at a certain position. -Although they may not do what you thought they did, C<\b> and C<\B> -can still be quite useful. For an example of the correct use of -C<\b>, see the example of matching duplicate words over multiple -lines. +For the regular expression, /\bPerl\b/, there has to be a word +boundary before the "P" and after the "l". As long as something other +than a word character precedes the "P" and succeeds the "l", the +pattern will match. These strings match /\bPerl\b/. -An example of using C<\B> is the pattern C<\Bis\B>. This will find -occurrences of "is" on the insides of words only, as in "thistle", but -not "this" or "island". + "Perl" # no word char before P or after l + "Perl " # same as previous (space is not a word char) + "'Perl'" # the ' char is not a word char + "Perl's" # no word char before P, non-word char after "l" -=head2 Why does using $&, $`, or $' slow my program down? +These strings do not match /\bPerl\b/. -Once Perl sees that you need one of these variables anywhere in -the program, it provides them on each and every pattern match. -The same mechanism that handles these provides for the use of $1, $2, -etc., so you pay the same price for each regex that contains capturing -parentheses. If you never use $&, etc., in your script, then regexes -I capturing parentheses won't be penalized. So avoid $&, $', -and $` if you can, but if you can't, once you've used them at all, use -them at will because you've already paid the price. Remember that some -algorithms really appreciate them. As of the 5.005 release. the $& -variable is no longer "expensive" the way the other two are. + "Perl_" # _ is a word char! + "Perler" # no word char before P, but one after l -=head2 What good is C<\G> in a regular expression? +You don't have to use \b to match words though. You can look for +non-word characters surrounded by word characters. These strings +match the pattern /\b'\b/. -The notation C<\G> is used in a match or substitution in conjunction with -the C modifier to anchor the regular expression to the point just past -where the last match occurred, i.e. the pos() point. A failed match resets -the position of C<\G> unless the C modifier is in effect. C<\G> can be -used in a match without the C modifier; it acts the same (i.e. still -anchors at the pos() point) but of course only matches once and does not -update pos(), as non-C expressions never do. C<\G> in an expression -applied to a target string that has never been matched against a C -expression before or has had its pos() reset is functionally equivalent to -C<\A>, which matches at the beginning of the string. + "don't" # the ' char is surrounded by "n" and "t" + "qep'a'" # the ' char is surrounded by "p" and "a" -For example, suppose you had a line of text quoted in standard mail -and Usenet notation, (that is, with leading C<< > >> characters), and -you want change each leading C<< > >> into a corresponding C<:>. You -could do so in this way: +These strings do not match /\b'\b/. - s/^(>+)/':' x length($1)/gem; + "foo'" # there is no word char after non-word ' -Or, using C<\G>, the much simpler (and faster): +You can also use the complement of \b, \B, to specify that there +should not be a word boundary. - s/\G>/:/g; +In the pattern /\Bam\B/, there must be a word character before the "a" +and after the "m". These patterns match /\Bam\B/: -A more sophisticated use might involve a tokenizer. The following -lex-like example is courtesy of Jeffrey Friedl. It did not work in -5.003 due to bugs in that release, but does work in 5.004 or better. -(Note the use of C, which prevents a failed match with C from -resetting the search position back to the beginning of the string.) + "llama" # "am" surrounded by word chars + "Samuel" # same - while (<>) { - chomp; - PARSER: { - m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; }; - m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; }; - m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; }; - m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; }; - } - } +These strings do not match /\Bam\B/ + + "Sam" # no word boundary before "a", but one after "m" + "I am Sam" # "am" surrounded by non-word chars + + +=head2 Why does using $&, $`, or $' slow my program down? -Of course, that could have been written as +(contributed by Anno Siegel) + +Once Perl sees that you need one of these variables anywhere in the +program, it provides them on each and every pattern match. That means +that on every pattern match the entire string will be copied, part of it +to $`, part to $&, and part to $'. Thus the penalty is most severe with +long strings and patterns that match often. Avoid $&, $', and $` if you +can, but if you can't, once you've used them at all, use them at will +because you've already paid the price. Remember that some algorithms +really appreciate them. As of the 5.005 release, the $& variable is no +longer "expensive" the way the other two are. + +Since Perl 5.6.1 the special variables @- and @+ can functionally replace +$`, $& and $'. These arrays contain pointers to the beginning and end +of each match (see perlvar for the full story), so they give you +essentially the same information, but without the risk of excessive +string copying. + +=head2 What good is C<\G> in a regular expression? + +You use the C<\G> anchor to start the next match on the same +string where the last match left off. The regular +expression engine cannot skip over any characters to find +the next match with this anchor, so C<\G> is similar to the +beginning of string anchor, C<^>. The C<\G> anchor is typically +used with the C flag. It uses the value of pos() +as the position to start the next match. As the match +operator makes successive matches, it updates pos() with the +position of the next character past the last match (or the +first character of the next match, depending on how you like +to look at it). Each string has its own pos() value. + +Suppose you want to match all of consective pairs of digits +in a string like "1122a44" and stop matching when you +encounter non-digits. You want to match C<11> and C<22> but +the letter shows up between C<22> and C<44> and you want +to stop at C. Simply matching pairs of digits skips over +the C and still matches C<44>. + + $_ = "1122a44"; + my @pairs = m/(\d\d)/g; # qw( 11 22 44 ) + +If you use the \G anchor, you force the match after C<22> to +start with the C. The regular expression cannot match +there since it does not find a digit, so the next match +fails and the match operator returns the pairs it already +found. + + $_ = "1122a44"; + my @pairs = m/\G(\d\d)/g; # qw( 11 22 ) + +You can also use the C<\G> anchor in scalar context. You +still need the C flag. + + $_ = "1122a44"; + while( m/\G(\d\d)/g ) + { + print "Found $1\n"; + } + +After the match fails at the letter C, perl resets pos() +and the next match on the same string starts at the beginning. + + $_ = "1122a44"; + while( m/\G(\d\d)/g ) + { + print "Found $1\n"; + } + + print "Found $1 after while" if m/(\d\d)/g; # finds "11" + +You can disable pos() resets on fail with the C flag. +Subsequent matches start where the last successful match +ended (the value of pos()) even if a match on the same +string as failed in the meantime. In this case, the match +after the while() loop starts at the C (where the last +match stopped), and since it does not use any anchor it can +skip over the C to find "44". + + $_ = "1122a44"; + while( m/\G(\d\d)/gc ) + { + print "Found $1\n"; + } + + print "Found $1 after while" if m/(\d\d)/g; # finds "44" + +Typically you use the C<\G> anchor with the C flag +when you want to try a different match if one fails, +such as in a tokenizer. Jeffrey Friedl offers this example +which works in 5.004 or later. while (<>) { chomp; PARSER: { - if ( /\G( \d+\b )/gcx { - print "number: $1\n"; - redo PARSER; - } - if ( /\G( \w+ )/gcx { - print "word: $1\n"; - redo PARSER; - } - if ( /\G( \s+ )/gcx { - print "space: $1\n"; - redo PARSER; - } - if ( /\G( [^\w\d]+ )/gcx { - print "other: $1\n"; - redo PARSER; - } + m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; }; + m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; }; + m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; }; + m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; }; } } -but then you lose the vertical alignment of the regular expressions. +For each line, the PARSER loop first tries to match a series +of digits followed by a word boundary. This match has to +start at the place the last match left off (or the beginning +of the string on the first match). Since C uses the C flag, if the string does not match that +regular expression, perl does not reset pos() and the next +match starts at the same position to try a different +pattern. =head2 Are Perl regexes DFAs or NFAs? Are they POSIX compliant? @@ -632,15 +754,18 @@ guaranteed is slowness.) See the book "Mastering Regular Expressions" hope to know on these matters (a full citation appears in L). -=head2 What's wrong with using grep or map in a void context? +=head2 What's wrong with using grep in a void context? -The problem is that both grep and map build a return list, -regardless of the context. This means you're making Perl go -to the trouble of building a list that you then just throw away. -If the list is large, you waste both time and space. If your -intent is to iterate over the list then use a for loop for this +The problem is that grep builds a return list, regardless of the context. +This means you're making Perl go to the trouble of building a list that +you then just throw away. If the list is large, you waste both time and space. +If your intent is to iterate over the list, then use a for loop for this purpose. +In perls older than 5.8.1, map suffers from this problem as well. +But since 5.8.1, this has been fixed, and map is context aware - in void +context, no lists are constructed. + =head2 How can I match strings with multibyte characters? Starting from Perl 5.6 Perl has had some level of multibyte character @@ -675,8 +800,8 @@ looks like it is because "SG" is next to "XX", but there's no real Here are a few ways, all painful, to deal with it: - $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian'' bytes - # are no longer adjacent. + $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent "martian" + # bytes are no longer adjacent. print "found GX!\n" if $martian =~ /GX/; Or like this: @@ -694,13 +819,21 @@ Or like this: print "found GX!\n", last if $1 eq 'GX'; } -Or like this: +Here's another, slightly less painful, way to do it from Benjamin +Goldberg, who uses a zero-width negative look-behind assertion. + + print "found GX!\n" if $martian =~ m/ + (?. =head1 AUTHOR AND COPYRIGHT -Copyright (c) 1997-2002 Tom Christiansen and Nathan Torkington. -All rights reserved. +Copyright (c) 1997-2005 Tom Christiansen, Nathan Torkington, and +other authors as noted. All rights reserved. This documentation is free; you can redistribute it and/or modify it under the same terms as Perl itself.