X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlfaq4.pod;h=6a882c53ff5b2c6718b0d0549ddd9a51b60ed1dc;hb=938c8732ceb115a707f725327a631eb35319ba87;hp=a5b505c4a7ab415d3cc9aac2d73b80c89e3c155a;hpb=2ceaccd71c2e15067691ff5d6509bed0b81708ee;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlfaq4.pod b/pod/perlfaq4.pod index a5b505c..6a882c5 100644 --- a/pod/perlfaq4.pod +++ b/pod/perlfaq4.pod @@ -1,71 +1,101 @@ =head1 NAME -perlfaq4 - Data Manipulation ($Revision: 1.19 $, $Date: 1997/04/24 22:43:57 $) +perlfaq4 - Data Manipulation ($Revision: 1.54 $, $Date: 2003/11/30 00:50:08 $) =head1 DESCRIPTION -The section of the FAQ answers question related to the manipulation -of data as numbers, dates, strings, arrays, hashes, and miscellaneous -data issues. +This section of the FAQ answers questions related to manipulating +numbers, dates, strings, arrays, hashes, and miscellaneous data issues. =head1 Data: Numbers =head2 Why am I getting long decimals (eg, 19.9499999999999) instead of the numbers I should be getting (eg, 19.95)? -Internally, your computer represents floating-point numbers in binary. -Floating-point numbers read in from a file, or appearing as literals -in your program, are converted from their decimal floating-point -representation (eg, 19.95) to the internal binary representation. +Internally, your computer represents floating-point numbers +in binary. Digital (as in powers of two) computers cannot +store all numbers exactly. Some real numbers lose precision +in the process. This is a problem with how computers store +numbers and affects all computer languages, not just Perl. -However, 19.95 can't be precisely represented as a binary -floating-point number, just like 1/3 can't be exactly represented as a -decimal floating-point number. The computer's binary representation -of 19.95, therefore, isn't exactly 19.95. +L show the gory details of number +representations and conversions. -When a floating-point number gets printed, the binary floating-point -representation is converted back to decimal. These decimal numbers -are displayed in either the format you specify with printf(), or the -current output format for numbers (see L if you use -print. C<$#> has a different default value in Perl5 than it did in -Perl4. Changing C<$#> yourself is deprecated. +To limit the number of decimal places in your numbers, you +can use the printf or sprintf function. See the +L<"Floating Point Arithmetic"|perlop> for more details. -This affects B computer languages that represent decimal -floating-point numbers in binary, not just Perl. Perl provides -arbitrary-precision decimal numbers with the Math::BigFloat module -(part of the standard Perl distribution), but mathematical operations -are consequently slower. + printf "%.2f", 10/3; -To get rid of the superfluous digits, just use a format (eg, -C) to get the required precision. + my $number = sprintf "%.2f", 10/3; + +=head2 Why is int() broken? + +Your int() is most probably working just fine. It's the numbers that +aren't quite what you think. + +First, see the above item "Why am I getting long decimals +(eg, 19.9499999999999) instead of the numbers I should be getting +(eg, 19.95)?". + +For example, this + + print int(0.6/0.2-2), "\n"; + +will in most computers print 0, not 1, because even such simple +numbers as 0.6 and 0.2 cannot be presented exactly by floating-point +numbers. What you think in the above as 'three' is really more like +2.9999999999999995559. =head2 Why isn't my octal data interpreted correctly? -Perl only understands octal and hex numbers as such when they occur -as literals in your program. If they are read in from somewhere and -assigned, no automatic conversion takes place. You must explicitly -use oct() or hex() if you want the values converted. oct() interprets -both hex ("0x350") numbers and octal ones ("0350" or even without the -leading "0", like "377"), while hex() only converts hexadecimal ones, -with or without a leading "0x", like "0x255", "3A", "ff", or "deadbeef". +Perl only understands octal and hex numbers as such when they occur as +literals in your program. Octal literals in perl must start with a +leading "0" and hexadecimal literals must start with a leading "0x". +If they are read in from somewhere and assigned, no automatic +conversion takes place. You must explicitly use oct() or hex() if you +want the values converted to decimal. oct() interprets hex ("0x350"), +octal ("0350" or even without the leading "0", like "377") and binary +("0b1010") numbers, while hex() only converts hexadecimal ones, with +or without a leading "0x", like "0x255", "3A", "ff", or "deadbeef". +The inverse mapping from decimal to octal can be done with either the +"%o" or "%O" sprintf() formats. This problem shows up most often when people try using chmod(), mkdir(), -umask(), or sysopen(), which all want permissions in octal. +umask(), or sysopen(), which by widespread tradition typically take +permissions in octal. - chmod(644, $file); # WRONG -- perl -w catches this + chmod(644, $file); # WRONG chmod(0644, $file); # right -=head2 Does perl have a round function? What about ceil() and floor()? -Trig functions? +Note the mistake in the first line was specifying the decimal literal +644, rather than the intended octal literal 0644. The problem can +be seen with: + + printf("%#o",644); # prints 01204 + +Surely you had not intended C - did you? If you +want to use numeric literals as arguments to chmod() et al. then please +try to express them as octal constants, that is with a leading zero and +with the following digits restricted to the set 0..7. -For rounding to a certain number of digits, sprintf() or printf() is -usually the easiest route. +=head2 Does Perl have a round() function? What about ceil() and floor()? Trig functions? -The POSIX module (part of the standard perl distribution) implements +Remember that int() merely truncates toward 0. For rounding to a +certain number of digits, sprintf() or printf() is usually the easiest +route. + + printf("%.3f", 3.1415926535); # prints 3.142 + +The POSIX module (part of the standard Perl distribution) implements ceil(), floor(), and a number of other mathematical and trigonometric functions. -In 5.000 to 5.003 Perls, trigonometry was done in the Math::Complex -module. With 5.004, the Math::Trig module (part of the standard perl + use POSIX; + $ceil = ceil(3.5); # 4 + $floor = floor(3.5); # 3 + +In 5.000 to 5.003 perls, trigonometry was done in the Math::Complex +module. With 5.004, the Math::Trig module (part of the standard Perl distribution) implements the trigonometric functions. Internally it uses the Math::Complex module and some functions can break out from the real axis into the complex plane, for example the inverse sine of @@ -77,17 +107,183 @@ cases, it probably pays not to trust whichever system rounding is being used by Perl, but to instead implement the rounding function you need yourself. -=head2 How do I convert bits into ints? +To see why, notice how you'll still have an issue on half-way-point +alternation: + + for ($i = 0; $i < 1.01; $i += 0.05) { printf "%.1f ",$i} + + 0.0 0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 0.6 0.7 0.7 + 0.8 0.8 0.9 0.9 1.0 1.0 + +Don't blame Perl. It's the same as in C. IEEE says we have to do this. +Perl numbers whose absolute values are integers under 2**31 (on 32 bit +machines) will work pretty much like mathematical integers. Other numbers +are not guaranteed. + +=head2 How do I convert between numeric representations/bases/radixes? + +As always with Perl there is more than one way to do it. Below +are a few examples of approaches to making common conversions +between number representations. This is intended to be representational +rather than exhaustive. + +Some of the examples below use the Bit::Vector module from CPAN. +The reason you might choose Bit::Vector over the perl built in +functions is that it works with numbers of ANY size, that it is +optimized for speed on some operations, and for at least some +programmers the notation might be familiar. + +=over 4 + +=item How do I convert hexadecimal into decimal + +Using perl's built in conversion of 0x notation: + + $dec = 0xDEADBEEF; + +Using the hex function: + + $dec = hex("DEADBEEF"); + +Using pack: + + $dec = unpack("N", pack("H8", substr("0" x 8 . "DEADBEEF", -8))); + +Using the CPAN module Bit::Vector: + + use Bit::Vector; + $vec = Bit::Vector->new_Hex(32, "DEADBEEF"); + $dec = $vec->to_Dec(); + +=item How do I convert from decimal to hexadecimal + +Using sprintf: + + $hex = sprintf("%X", 3735928559); # upper case A-F + $hex = sprintf("%x", 3735928559); # lower case a-f + +Using unpack: + + $hex = unpack("H*", pack("N", 3735928559)); + +Using Bit::Vector: + + use Bit::Vector; + $vec = Bit::Vector->new_Dec(32, -559038737); + $hex = $vec->to_Hex(); + +And Bit::Vector supports odd bit counts: + + use Bit::Vector; + $vec = Bit::Vector->new_Dec(33, 3735928559); + $vec->Resize(32); # suppress leading 0 if unwanted + $hex = $vec->to_Hex(); + +=item How do I convert from octal to decimal + +Using Perl's built in conversion of numbers with leading zeros: + + $dec = 033653337357; # note the leading 0! + +Using the oct function: + + $dec = oct("33653337357"); + +Using Bit::Vector: + + use Bit::Vector; + $vec = Bit::Vector->new(32); + $vec->Chunk_List_Store(3, split(//, reverse "33653337357")); + $dec = $vec->to_Dec(); + +=item How do I convert from decimal to octal + +Using sprintf: + + $oct = sprintf("%o", 3735928559); + +Using Bit::Vector: + + use Bit::Vector; + $vec = Bit::Vector->new_Dec(32, -559038737); + $oct = reverse join('', $vec->Chunk_List_Read(3)); + +=item How do I convert from binary to decimal + +Perl 5.6 lets you write binary numbers directly with +the 0b notation: + + $number = 0b10110110; + +Using oct: + + my $input = "10110110"; + $decimal = oct( "0b$input" ); + +Using pack and ord: + + $decimal = ord(pack('B8', '10110110')); + +Using pack and unpack for larger strings: + + $int = unpack("N", pack("B32", + substr("0" x 32 . "11110101011011011111011101111", -32))); + $dec = sprintf("%d", $int); + + # substr() is used to left pad a 32 character string with zeros. + +Using Bit::Vector: + + $vec = Bit::Vector->new_Bin(32, "11011110101011011011111011101111"); + $dec = $vec->to_Dec(); + +=item How do I convert from decimal to binary + +Using sprintf (perl 5.6+): -To turn a string of 1s and 0s like '10110110' into a scalar containing -its binary value, use the pack() function (documented in -L): + $bin = sprintf("%b", 3735928559); - $decimal = pack('B8', '10110110'); +Using unpack: -Here's an example of going the other way: + $bin = unpack("B*", pack("N", 3735928559)); - $binary_string = join('', unpack('B*', "\x29")); +Using Bit::Vector: + + use Bit::Vector; + $vec = Bit::Vector->new_Dec(32, -559038737); + $bin = $vec->to_Bin(); + +The remaining transformations (e.g. hex -> oct, bin -> hex, etc.) +are left as an exercise to the inclined reader. + +=back + +=head2 Why doesn't & work the way I want it to? + +The behavior of binary arithmetic operators depends on whether they're +used on numbers or strings. The operators treat a string as a series +of bits and work with that (the string C<"3"> is the bit pattern +C<00110011>). The operators work with the binary form of a number +(the number C<3> is treated as the bit pattern C<00000011>). + +So, saying C<11 & 3> performs the "and" operation on numbers (yielding +C<3>). Saying C<"11" & "3"> performs the "and" operation on strings +(yielding C<"1">). + +Most problems with C<&> and C<|> arise because the programmer thinks +they have a number but really it's a string. The rest arise because +the programmer says: + + if ("\020\020" & "\101\101") { + # ... + } + +but a string consisting of two null bytes (the result of C<"\020\020" +& "\101\101">) is not a false value in Perl. You need: + + if ( ("\020\020" & "\101\101") !~ /[^\000]/) { + # ... + } =head2 How do I multiply matrices? @@ -109,12 +305,12 @@ To call a function on each element of an array, but ignore the results: foreach $iterator (@array) { - &my_func($iterator); + some_func($iterator); } To call a function on each integer in a (small) range, you B use: - @results = map { &my_func($_) } (5 .. 25); + @results = map { some_func($_) } (5 .. 25); but you should be aware that the C<..> operator creates an array of all integers in the range. This can take a lot of memory for large @@ -122,139 +318,342 @@ ranges. Instead use: @results = (); for ($i=5; $i < 500_005; $i++) { - push(@results, &my_func($i)); + push(@results, some_func($i)); + } + +This situation has been fixed in Perl5.005. Use of C<..> in a C +loop will iterate over the range, without creating the entire range. + + for my $i (5 .. 500_005) { + push(@results, some_func($i)); } +will not create a list of 500,000 integers. + =head2 How can I output Roman numerals? -Get the http://www.perl.com/CPAN/modules/by-module/Roman module. +Get the http://www.cpan.org/modules/by-module/Roman module. =head2 Why aren't my random numbers random? -The short explanation is that you're getting pseudorandom numbers, not -random ones, because that's how these things work. A longer -explanation is available on -http://www.perl.com/CPAN/doc/FMTEYEWTK/random, courtesy of Tom -Phoenix. +If you're using a version of Perl before 5.004, you must call C +once at the start of your program to seed the random number generator. + + BEGIN { srand() if $] < 5.004 } + +5.004 and later automatically call C at the beginning. Don't +call C more than once---you make your numbers less random, rather +than more. + +Computers are good at being predictable and bad at being random +(despite appearances caused by bugs in your programs :-). see the +F article in the "Far More Than You Ever Wanted To Know" +collection in http://www.cpan.org/misc/olddoc/FMTEYEWTK.tgz , courtesy of +Tom Phoenix, talks more about this. John von Neumann said, ``Anyone +who attempts to generate random numbers by deterministic means is, of +course, living in a state of sin.'' + +If you want numbers that are more random than C with C +provides, you should also check out the Math::TrulyRandom module from +CPAN. It uses the imperfections in your system's timer to generate +random numbers, but this takes quite a while. If you want a better +pseudorandom generator than comes with your operating system, look at +``Numerical Recipes in C'' at http://www.nr.com/ . + +=head2 How do I get a random number between X and Y? + +C returns a number such that +C<< 0 <= rand($x) < $x >>. Thus what you want to have perl +figure out is a random number in the range from 0 to the +difference between your I and I. + +That is, to get a number between 10 and 15, inclusive, you +want a random number between 0 and 5 that you can then add +to 10. -You should also check out the Math::TrulyRandom module from CPAN. + my $number = 10 + int rand( 15-10+1 ); + +Hence you derive the following simple function to abstract +that. It selects a random integer between the two given +integers (inclusive), For example: C. + + sub random_int_in ($$) { + my($min, $max) = @_; + # Assumes that the two arguments are integers themselves! + return $min if $min == $max; + ($min, $max) = ($max, $min) if $min > $max; + return $min + int rand(1 + $max - $min); + } =head1 Data: Dates -=head2 How do I find the week-of-the-year/day-of-the-year? +=head2 How do I find the day or week of the year? -The day of the year is in the array returned by localtime() (see -L): +The localtime function returns the day of the week. Without an +argument localtime uses the current time. - $day_of_year = (localtime(time()))[7]; + $day_of_year = (localtime)[7]; -or more legibly (in 5.004 or higher): +The POSIX module can also format a date as the day of the year or +week of the year. - use Time::localtime; - $day_of_year = localtime(time())->yday; + use POSIX qw/strftime/; + my $day_of_year = strftime "%j", localtime; + my $week_of_year = strftime "%W", localtime; -You can find the week of the year by dividing this by 7: +To get the day of year for any date, use the Time::Local module to get +a time in epoch seconds for the argument to localtime. - $week_of_year = int($day_of_year / 7); + use POSIX qw/strftime/; + use Time::Local; + my $week_of_year = strftime "%W", + localtime( timelocal( 0, 0, 0, 18, 11, 1987 ) ); -Of course, this believes that weeks start at zero. +The Date::Calc module provides two functions for to calculate these. -=head2 How can I compare two date strings? + use Date::Calc; + my $day_of_year = Day_of_Year( 1987, 12, 18 ); + my $week_of_year = Week_of_Year( 1987, 12, 18 ); -Use the Date::Manip or Date::DateCalc modules from CPAN. +=head2 How do I find the current century or millennium? -=head2 How can I take a string and turn it into epoch seconds? +Use the following simple functions: -If it's a regular enough string that it always has the same format, -you can split it up and pass the parts to timelocal in the standard -Time::Local module. Otherwise, you should look into one of the -Date modules from CPAN. + sub get_century { + return int((((localtime(shift || time))[5] + 1999))/100); + } + sub get_millennium { + return 1+int((((localtime(shift || time))[5] + 1899))/1000); + } -=head2 How can I find the Julian Day? +On some systems, the POSIX module's strftime() function has +been extended in a non-standard way to use a C<%C> format, +which they sometimes claim is the "century". It isn't, +because on most such systems, this is only the first two +digits of the four-digit year, and thus cannot be used to +reliably determine the current century or millennium. + +=head2 How can I compare two dates and find the difference? + +If you're storing your dates as epoch seconds then simply subtract one +from the other. If you've got a structured date (distinct year, day, +month, hour, minute, seconds values), then for reasons of accessibility, +simplicity, and efficiency, merely use either timelocal or timegm (from +the Time::Local module in the standard distribution) to reduce structured +dates to epoch seconds. However, if you don't know the precise format of +your dates, then you should probably use either of the Date::Manip and +Date::Calc modules from CPAN before you go hacking up your own parsing +routine to handle arbitrary date formats. -Neither Date::Manip nor Date::DateCalc deal with Julian days. -Instead, there is an example of Julian date calculation in -http://www.perl.com/CPAN/authors/David_Muir_Sharnoff/modules/Time/JulianDay.pm.gz, -which should help. +=head2 How can I take a string and turn it into epoch seconds? -=head2 Does Perl have a year 2000 problem? +If it's a regular enough string that it always has the same format, +you can split it up and pass the parts to C in the standard +Time::Local module. Otherwise, you should look into the Date::Calc +and Date::Manip modules from CPAN. -Not unless you use Perl to create one. The date and time functions -supplied with perl (gmtime and localtime) supply adequate information -to determine the year well beyond 2000 (2038 is when trouble strikes). -The year returned by these functions when used in an array context is -the year minus 1900. For years between 1910 and 1999 this I -to be a 2-digit decimal number. To avoid the year 2000 problem simply -do not treat the year as a 2-digit number. It isn't. +=head2 How can I find the Julian Day? -When gmtime() and localtime() are used in a scalar context they return +Use the Time::JulianDay module (part of the Time-modules bundle +available from CPAN.) + +Before you immerse yourself too deeply in this, be sure to verify that +it is the I Day you really want. Are you interested in a way +of getting serial days so that you just can tell how many days they +are apart or so that you can do also other date arithmetic? If you +are interested in performing date arithmetic, this can be done using +modules Date::Manip or Date::Calc. + +There is too many details and much confusion on this issue to cover in +this FAQ, but the term is applied (correctly) to a calendar now +supplanted by the Gregorian Calendar, with the Julian Calendar failing +to adjust properly for leap years on centennial years (among other +annoyances). The term is also used (incorrectly) to mean: [1] days in +the Gregorian Calendar; and [2] days since a particular starting time +or `epoch', usually 1970 in the Unix world and 1980 in the +MS-DOS/Windows world. If you find that it is not the first meaning +that you really want, then check out the Date::Manip and Date::Calc +modules. (Thanks to David Cassell for most of this text.) + +=head2 How do I find yesterday's date? + +If you only need to find the date (and not the same time), you +can use the Date::Calc module. + + use Date::Calc qw(Today Add_Delta_Days); + + my @date = Add_Delta_Days( Today(), -1 ); + + print "@date\n"; + +Most people try to use the time rather than the calendar to +figure out dates, but that assumes that your days are +twenty-four hours each. For most people, there are two days +a year when they aren't: the switch to and from summer time +throws this off. Russ Allbery offers this solution. + + sub yesterday { + my $now = defined $_[0] ? $_[0] : time; + my $then = $now - 60 * 60 * 24; + my $ndst = (localtime $now)[8] > 0; + my $tdst = (localtime $then)[8] > 0; + $then - ($tdst - $ndst) * 60 * 60; + } + +Should give you "this time yesterday" in seconds since epoch relative to +the first argument or the current time if no argument is given and +suitable for passing to localtime or whatever else you need to do with +it. $ndst is whether we're currently in daylight savings time; $tdst is +whether the point 24 hours ago was in daylight savings time. If $tdst +and $ndst are the same, a boundary wasn't crossed, and the correction +will subtract 0. If $tdst is 1 and $ndst is 0, subtract an hour more +from yesterday's time since we gained an extra hour while going off +daylight savings time. If $tdst is 0 and $ndst is 1, subtract a +negative hour (add an hour) to yesterday's time since we lost an hour. + +All of this is because during those days when one switches off or onto +DST, a "day" isn't 24 hours long; it's either 23 or 25. + +The explicit settings of $ndst and $tdst are necessary because localtime +only says it returns the system tm struct, and the system tm struct at +least on Solaris doesn't guarantee any particular positive value (like, +say, 1) for isdst, just a positive value. And that value can +potentially be negative, if DST information isn't available (this sub +just treats those cases like no DST). + +Note that between 2am and 3am on the day after the time zone switches +off daylight savings time, the exact hour of "yesterday" corresponding +to the current hour is not clearly defined. Note also that if used +between 2am and 3am the day after the change to daylight savings time, +the result will be between 3am and 4am of the previous day; it's +arguable whether this is correct. + +This sub does not attempt to deal with leap seconds (most things don't). + + + +=head2 Does Perl have a Year 2000 problem? Is Perl Y2K compliant? + +Short answer: No, Perl does not have a Year 2000 problem. Yes, Perl is +Y2K compliant (whatever that means). The programmers you've hired to +use it, however, probably are not. + +Long answer: The question belies a true understanding of the issue. +Perl is just as Y2K compliant as your pencil--no more, and no less. +Can you use your pencil to write a non-Y2K-compliant memo? Of course +you can. Is that the pencil's fault? Of course it isn't. + +The date and time functions supplied with Perl (gmtime and localtime) +supply adequate information to determine the year well beyond 2000 +(2038 is when trouble strikes for 32-bit machines). The year returned +by these functions when used in a list context is the year minus 1900. +For years between 1910 and 1999 this I to be a 2-digit decimal +number. To avoid the year 2000 problem simply do not treat the year as +a 2-digit number. It isn't. + +When gmtime() and localtime() are used in scalar context they return a timestamp string that contains a fully-expanded year. For example, C<$timestamp = gmtime(1005613200)> sets $timestamp to "Tue Nov 13 01:00:00 2001". There's no year 2000 problem here. +That doesn't mean that Perl can't be used to create non-Y2K compliant +programs. It can. But so can your pencil. It's the fault of the user, +not the language. At the risk of inflaming the NRA: ``Perl doesn't +break Y2K, people do.'' See http://language.perl.com/news/y2k.html for +a longer exposition. + =head1 Data: Strings =head2 How do I validate input? The answer to this question is usually a regular expression, perhaps -with auxiliary logic. See the more specific questions (numbers, email +with auxiliary logic. See the more specific questions (numbers, mail addresses, etc.) for details. =head2 How do I unescape a string? -It depends just what you mean by "escape". URL escapes are dealt with -in L. Shell escapes with the backslash (\) -character are removed with: +It depends just what you mean by ``escape''. URL escapes are dealt +with in L. Shell escapes with the backslash (C<\>) +character are removed with s/\\(.)/$1/g; -Note that this won't expand \n or \t or any other special escapes. +This won't expand C<"\n"> or C<"\t"> or any other special escapes. =head2 How do I remove consecutive pairs of characters? -To turn "abbcccd" into "abccd": +To turn C<"abbcccd"> into C<"abccd">: + + s/(.)\1/$1/g; # add /s to include newlines - s/(.)\1/$1/g; +Here's a solution that turns "abbcccd" to "abcd": + + y///cs; # y == tr, but shorter :-) =head2 How do I expand function calls in a string? This is documented in L. In general, this is fraught with quoting and readability problems, but it is possible. To interpolate -a subroutine call (in a list context) into a string: +a subroutine call (in list context) into a string: print "My sub returned @{[mysub(1,2,3)]} that time.\n"; -If you prefer scalar context, similar chicanery is also useful for -arbitrary expressions: +See also ``How can I expand variables in text strings?'' in this +section of the FAQ. - print "That yields ${\($n + 5)} widgets\n"; +=head2 How do I find matching/nesting anything? -See also "How can I expand variables in text strings?" in this section -of the FAQ. +This isn't something that can be done in one regular expression, no +matter how complicated. To find something between two single +characters, a pattern like C will get the intervening +bits in $1. For multiple ones, then something more like +C would be needed. But none of these deals with +nested patterns. For balanced expressions using C<(>, C<{>, C<[> +or C<< < >> as delimiters, use the CPAN module Regexp::Common, or see +L. For other cases, you'll have to write a parser. + +If you are serious about writing a parser, there are a number of +modules or oddities that will make your life a lot easier. There are +the CPAN modules Parse::RecDescent, Parse::Yapp, and Text::Balanced; +and the byacc program. Starting from perl 5.8 the Text::Balanced +is part of the standard distribution. + +One simple destructive, inside-out approach that you might try is to +pull out the smallest nesting parts one at a time: + + while (s/BEGIN((?:(?!BEGIN)(?!END).)*)END//gs) { + # do something with $1 + } -=head2 How do I find matching/nesting anything? +A more complicated and sneaky approach is to make Perl's regular +expression engine do it for you. This is courtesy Dean Inada, and +rather has the nature of an Obfuscated Perl Contest entry, but it +really does work: + + # $_ contains the string to parse + # BEGIN and END are the opening and closing markers for the + # nested text. -This isn't something that can be tackled in one regular expression, no -matter how complicated. To find something between two single characters, -a pattern like C will get the intervening bits in $1. For -multiple ones, then something more like C would -be needed. But none of these deals with nested patterns, nor can they. -For that you'll have to write a parser. + @( = ('(',''); + @) = (')',''); + ($re=$_)=~s/((BEGIN)|(END)|.)/$)[!$3]\Q$1\E$([!$2]/gs; + @$ = (eval{/$re/},$@!~/unmatched/i); + print join("\n",@$[0..$#$]) if( $$[-1] ); =head2 How do I reverse a string? -Use reverse() in a scalar context, as documented in +Use reverse() in scalar context, as documented in L. $reversed = reverse $string; =head2 How do I expand tabs in a string? -You can do it the old-fashioned way: +You can do it yourself: 1 while $string =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e; -Or you can just use the Text::Tabs module (part of the standard perl +Or you can just use the Text::Tabs module (part of the standard Perl distribution). use Text::Tabs; @@ -262,52 +661,82 @@ distribution). =head2 How do I reformat a paragraph? -Use Text::Wrap (part of the standard perl distribution): +Use Text::Wrap (part of the standard Perl distribution): use Text::Wrap; print wrap("\t", ' ', @paragraphs); -The paragraphs you give to Text::Wrap may not contain embedded +The paragraphs you give to Text::Wrap should not contain embedded newlines. Text::Wrap doesn't justify the lines (flush-right). -=head2 How can I access/change the first N letters of a string? +Or use the CPAN module Text::Autoformat. Formatting files can be easily +done by making a shell alias, like so: + + alias fmt="perl -i -MText::Autoformat -n0777 \ + -e 'print autoformat $_, {all=>1}' $*" + +See the documentation for Text::Autoformat to appreciate its many +capabilities. -There are many ways. If you just want to grab a copy, use -substr: +=head2 How can I access or change N characters of a string? - $first_byte = substr($a, 0, 1); +You can access the first characters of a string with substr(). +To get the first character, for example, start at position 0 +and grab the string of length 1. -If you want to modify part of a string, the simplest way is often to -use substr() as an lvalue: - substr($a, 0, 3) = "Tom"; + $string = "Just another Perl Hacker"; + $first_char = substr( $string, 0, 1 ); # 'J' -Although those with a regexp kind of thought process will likely prefer +To change part of a string, you can use the optional fourth +argument which is the replacement string. - $a =~ s/^.../Tom/; + substr( $string, 13, 4, "Perl 5.8.0" ); + +You can also use substr() as an lvalue. + + substr( $string, 13, 4 ) = "Perl 5.8.0"; =head2 How do I change the Nth occurrence of something? -You have to keep track. For example, let's say you want -to change the fifth occurrence of "whoever" or "whomever" -into "whosoever" or "whomsoever", case insensitively. +You have to keep track of N yourself. For example, let's say you want +to change the fifth occurrence of C<"whoever"> or C<"whomever"> into +C<"whosoever"> or C<"whomsoever">, case insensitively. These +all assume that $_ contains the string to be altered. $count = 0; s{((whom?)ever)}{ ++$count == 5 # is it the 5th? ? "${2}soever" # yes, swap : $1 # renege and leave it there - }igex; + }ige; + +In the more general case, you can use the C modifier in a C +loop, keeping count of matches. + + $WANT = 3; + $count = 0; + $_ = "One fish two fish red fish blue fish"; + while (/(\w+)\s+fish\b/gi) { + if (++$count == $WANT) { + print "The third fish is a $1 one.\n"; + } + } + +That prints out: C<"The third fish is a red one."> You can also use a +repetition count and repeated pattern like this: + + /(?:\w+\s+fish\s+){2}(\w+)\s+fish/i; =head2 How can I count the number of occurrences of a substring within a string? -There are a number of ways, with varying efficiency: If you want a +There are a number of ways, with varying efficiency. If you want a count of a certain single character (X) within a string, you can use the C function like so: - $string = "ThisXlineXhasXsomeXx'sXinXit": + $string = "ThisXlineXhasXsomeXx'sXinXit"; $count = ($string =~ tr/X//); - print "There are $count X charcters in the string"; + print "There are $count X characters in the string"; This is fine if you are just looking for a single character. However, if you are trying to count multiple character substrings within a @@ -319,6 +748,11 @@ integers: while ($string =~ /-\d+/g) { $count++ } print "There are $count negative numbers in the string"; +Another version uses a global match in list context, then assigns the +result to a scalar, producing a count of the number of matches. + + $count = () = $string =~ /-\d+/g; + =head2 How do I capitalize all the words on one line? To make the first letter of each word upper case: @@ -326,8 +760,8 @@ To make the first letter of each word upper case: $line =~ s/\b(\w)/\U$1/g; This has the strange effect of turning "C" into "C". Sometimes you might want this, instead (Suggested by Brian -Foy Ecomdog@computerdog.comE): +Do It>". Sometimes you might want this. Other times you might need a +more thorough solution (Suggested by brian d foy): $string =~ s/ ( (^\w) #at the beginning of the line @@ -345,20 +779,43 @@ To force each word to be lower case, with the first letter upper case: $line =~ s/(\w+)/\u\L$1/g; -=head2 How can I split a [character] delimited string except when inside -[character]? (Comma-separated files) +You can (and probably should) enable locale awareness of those +characters by placing a C pragma in your program. +See L for endless details on locales. + +This is sometimes referred to as putting something into "title +case", but that's not quite accurate. Consider the proper +capitalization of the movie I, for example. -Take the example case of trying to split a string that is comma-separated -into its different fields. (We'll pretend you said comma-separated, not -comma-delimited, which is different and almost never what you mean.) You -can't use C because you shouldn't split if the comma is inside -quotes. For example, take a data line like this: +Damian Conway's L module provides some smart +case transformations: + + use Text::Autoformat; + my $x = "Dr. Strangelove or: How I Learned to Stop ". + "Worrying and Love the Bomb"; + + print $x, "\n"; + for my $style (qw( sentence title highlight )) + { + print autoformat($x, { case => $style }), "\n"; + } + +=head2 How can I split a [character] delimited string except when inside [character]? + +Several modules can handle this sort of pasing---Text::Balanced, +Text::CVS, Text::CVS_XS, and Text::ParseWords, among others. + +Take the example case of trying to split a string that is +comma-separated into its different fields. You can't use C +because you shouldn't split if the comma is inside quotes. For +example, take a data line like this: SAR001,"","Cimetrix, Inc","Bob Smith","CAM",N,8,1,0,7,"Error, Core Dumped" Due to the restriction of the quotes, this is a fairly complex -problem. Thankfully, we have Jeffrey Friedl, author of a highly -recommended book on regular expressions, to handle these for us. He +problem. Thankfully, we have Jeffrey Friedl, author of +I, to handle these for us. He suggests (assuming your string is contained in $text): @new = (); @@ -371,22 +828,24 @@ suggests (assuming your string is contained in $text): If you want to represent quotation marks inside a quotation-mark-delimited field, escape them with backslashes (eg, -C<"like \"this\"">. Unescaping them is a task addressed earlier in -this section. +C<"like \"this\"">. -Alternatively, the Text::ParseWords module (part of the standard perl +Alternatively, the Text::ParseWords module (part of the standard Perl distribution) lets you say: use Text::ParseWords; @new = quotewords(",", 0, $text); +There's also a Text::CSV (Comma-Separated Values) module on CPAN. + =head2 How do I strip blank space from the beginning/end of a string? -The simplest approach, albeit not the fastest, is probably like this: +Although the simplest approach would seem to be $string =~ s/^\s*(.*?)\s*$/$1/; -It would be faster to do this in two steps: +not only is this unnecessarily slow and destructive, it also fails with +embedded newlines. It is much faster to do this operation in two steps: $string =~ s/^\s+//; $string =~ s/\s+$//; @@ -398,37 +857,132 @@ Or more nicely written as: s/\s+$//; } +This idiom takes advantage of the C loop's aliasing +behavior to factor out common code. You can do this +on several strings at once, or arrays, or even the +values of a hash if you use a slice: + + # trim whitespace in the scalar, the array, + # and all the values in the hash + foreach ($scalar, @array, @hash{keys %hash}) { + s/^\s+//; + s/\s+$//; + } + +=head2 How do I pad a string with blanks or pad a number with zeroes? + +In the following examples, C<$pad_len> is the length to which you wish +to pad the string, C<$text> or C<$num> contains the string to be padded, +and C<$pad_char> contains the padding character. You can use a single +character string constant instead of the C<$pad_char> variable if you +know what it is in advance. And in the same way you can use an integer in +place of C<$pad_len> if you know the pad length in advance. + +The simplest method uses the C function. It can pad on the left +or right with blanks and on the left with zeroes and it will not +truncate the result. The C function can only pad strings on the +right with blanks and it will truncate the result to a maximum length of +C<$pad_len>. + + # Left padding a string with blanks (no truncation): + $padded = sprintf("%${pad_len}s", $text); + $padded = sprintf("%*s", $pad_len, $text); # same thing + + # Right padding a string with blanks (no truncation): + $padded = sprintf("%-${pad_len}s", $text); + $padded = sprintf("%-*s", $pad_len, $text); # same thing + + # Left padding a number with 0 (no truncation): + $padded = sprintf("%0${pad_len}d", $num); + $padded = sprintf("%0*d", $pad_len, $num); # same thing + + # Right padding a string with blanks using pack (will truncate): + $padded = pack("A$pad_len",$text); + +If you need to pad with a character other than blank or zero you can use +one of the following methods. They all generate a pad string with the +C operator and combine that with C<$text>. These methods do +not truncate C<$text>. + +Left and right padding with any character, creating a new string: + + $padded = $pad_char x ( $pad_len - length( $text ) ) . $text; + $padded = $text . $pad_char x ( $pad_len - length( $text ) ); + +Left and right padding with any character, modifying C<$text> directly: + + substr( $text, 0, 0 ) = $pad_char x ( $pad_len - length( $text ) ); + $text .= $pad_char x ( $pad_len - length( $text ) ); + =head2 How do I extract selected columns from a string? Use substr() or unpack(), both documented in L. +If you prefer thinking in terms of columns instead of widths, +you can use this kind of thing: + + # determine the unpack format needed to split Linux ps output + # arguments are cut columns + my $fmt = cut2fmt(8, 14, 20, 26, 30, 34, 41, 47, 59, 63, 67, 72); + + sub cut2fmt { + my(@positions) = @_; + my $template = ''; + my $lastpos = 1; + for my $place (@positions) { + $template .= "A" . ($place - $lastpos) . " "; + $lastpos = $place; + } + $template .= "A*"; + return $template; + } =head2 How do I find the soundex value of a string? -Use the standard Text::Soundex module distributed with perl. +Use the standard Text::Soundex module distributed with Perl. +Before you do so, you may want to determine whether `soundex' is in +fact what you think it is. Knuth's soundex algorithm compresses words +into a small space, and so it does not necessarily distinguish between +two words which you might want to appear separately. For example, the +last names `Knuth' and `Kant' are both mapped to the soundex code K530. +If Text::Soundex does not do what you are looking for, you might want +to consider the String::Approx module available at CPAN. =head2 How can I expand variables in text strings? Let's assume that you have a string like: $text = 'this has a $foo in it and a $bar'; - $text =~ s/\$(\w+)/${$1}/g; -Before version 5 of perl, this had to be done with a double-eval -substitution: +If those were both global variables, then this would +suffice: + + $text =~ s/\$(\w+)/${$1}/g; # no /e needed + +But since they are probably lexicals, or at least, they could +be, you'd have to do this: $text =~ s/(\$\w+)/$1/eeg; + die if $@; # needed /ee, not /e + +It's probably better in the general case to treat those +variables as entries in some special hash. For example: -Which is bizarre enough that you'll probably actually need an EEG -afterwards. :-) + %user_defs = ( + foo => 23, + bar => 19, + ); + $text =~ s/\$(\w+)/$user_defs{$1}/g; -See also "How do I expand function calls in a string?" in this section +See also ``How do I expand function calls in a string?'' in this section of the FAQ. =head2 What's wrong with always quoting "$vars"? -The problem is that those double-quotes force stringification, -coercing numbers and references into strings, even when you -don't want them to be. +The problem is that those double-quotes force stringification-- +coercing numbers and references into strings--even when you +don't want them to be strings. Think of it this way: double-quote +expansion is used to produce new strings. If you already +have a string, why do you need more? If you get used to writing odd things like these: @@ -458,25 +1012,118 @@ that actually do care about the difference between a string and a number, such as the magical C<++> autoincrement operator or the syscall() function. -=head2 Why don't my <EHERE documents work? Check for these three things: =over 4 -=item 1. There must be no space after the << part. +=item There must be no space after the EE part. -=item 2. There (probably) should be a semicolon at the end. +=item There (probably) should be a semicolon at the end. -=item 3. You can't (easily) have any space in front of the tag. +=item You can't (easily) have any space in front of the tag. =back +If you want to indent the text in the here document, you +can do this: + + # all in one + ($VAR = <op_ppaddr)() ); + @@@ TAINT_NOT; + @@@ return 0; + @@@ } + MAIN_INTERPRETER_LOOP + +Or with a fixed amount of leading whitespace, with remaining +indentation correctly preserved: + + $poem = fix< variables are arrays, anonymous arrays are arrays, arrays +in scalar context behave like the number of elements in them, subroutines +access their arguments through the array C<@_>, and push/pop/shift only work +on arrays. + +As a side note, there's no such thing as a list in scalar context. +When you say + + $scalar = (2, 5, 7, 9); + +you're using the comma operator in scalar context, so it uses the scalar +comma operator. There never was a list there at all! This causes the +last value to be returned: 9. + =head2 What is the difference between $array[1] and @array[1]? -The former is a scalar value, the latter an array slice, which makes +The former is a scalar value; the latter an array slice, making it a list with one (scalar) value. You should use $ when you want a scalar value (most of the time) and @ when you want a list with one scalar value in it (very, very rarely; nearly never, in fact). @@ -490,56 +1137,75 @@ with @bad[0] = `same program that outputs several lines`; -The B<-w> flag will warn you about these matters. +The C pragma and the B<-w> flag will warn you about these +matters. -=head2 How can I extract just the unique elements of an array? +=head2 How can I remove duplicate elements from a list or array? There are several possible ways, depending on whether the array is ordered and whether you wish to preserve the ordering. =over 4 -=item a) If @in is sorted, and you want @out to be sorted: +=item a) + +If @in is sorted, and you want @out to be sorted: +(this assumes all true values in the array) - $prev = 'nonesuch'; - @out = grep($_ ne $prev && ($prev = $_), @in); + $prev = "not equal to $in[0]"; + @out = grep($_ ne $prev && ($prev = $_, 1), @in); -This is nice in that it doesn't use much extra memory, -simulating uniq(1)'s behavior of removing only adjacent -duplicates. +This is nice in that it doesn't use much extra memory, simulating +uniq(1)'s behavior of removing only adjacent duplicates. The ", 1" +guarantees that the expression is true (so that grep picks it up) +even if the $_ is 0, "", or undef. -=item b) If you don't know whether @in is sorted: +=item b) + +If you don't know whether @in is sorted: undef %saw; @out = grep(!$saw{$_}++, @in); -=item c) Like (b), but @in contains only small integers: +=item c) + +Like (b), but @in contains only small integers: @out = grep(!$saw[$_]++, @in); -=item d) A way to do (b) without any loops or greps: +=item d) + +A way to do (b) without any loops or greps: undef %saw; @saw{@in} = (); @out = sort keys %saw; # remove sort if undesired -=item e) Like (d), but @in contains only small positive integers: +=item e) + +Like (d), but @in contains only small positive integers: undef @ary; @ary[@in] = @in; - @out = @ary; + @out = grep {defined} @ary; =back -=head2 How can I tell whether an array contains a certain element? +But perhaps you should have been using a hash all along, eh? -There are several ways to approach this. If you are going to make -this query many times and the values are arbitrary strings, the -fastest way is probably to invert the original array and keep an -associative array lying about whose keys are the first array's values. +=head2 How can I tell whether a certain element is contained in a list or array? + +Hearing the word "in" is an Idication that you probably should have +used a hash, not a list or array, to store your data. Hashes are +designed to answer this question quickly and efficiently. Arrays aren't. + +That being said, there are several ways to approach this. If you +are going to make this query many times over arbitrary string values, +the fastest way is probably to invert the original array and maintain a +hash whose keys are the first array's values. @blues = qw/azure cerulean teal turquoise lapis-lazuli/; - undef %is_blue; + %is_blue = (); for (@blues) { $is_blue{$_} = 1 } Now you can check whether $is_blue{$some_color}. It might have been a @@ -549,8 +1215,9 @@ If the values are all small integers, you could use a simple indexed array. This kind of an array will take up less space: @primes = (2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31); - undef @is_tiny_prime; - for (@primes) { $is_tiny_prime[$_] = 1; } + @is_tiny_prime = (); + for (@primes) { $is_tiny_prime[$_] = 1 } + # or simply @istiny_prime[@primes] = (1) x @primes; Now you check whether $is_tiny_prime[$some_number]. @@ -559,21 +1226,31 @@ quite a lot of space by using bit strings instead: @articles = ( 1..10, 150..2000, 2017 ); undef $read; - grep (vec($read,$_,1) = 1, @articles); + for (@articles) { vec($read,$_,1) = 1 } Now check whether C is true for some C<$n>. Please do not use - $is_there = grep $_ eq $whatever, @array; + ($is_there) = grep $_ eq $whatever, @array; or worse yet - $is_there = grep /$whatever/, @array; + ($is_there) = grep /$whatever/, @array; These are slow (checks every element even if the first matches), inefficient (same reason), and potentially buggy (what if there are -regexp characters in $whatever?). +regex characters in $whatever?). If you're only testing once, then +use: + + $is_there = 0; + foreach $elt (@array) { + if ($elt eq $elt_to_find) { + $is_there = 1; + last; + } + } + if ($is_there) { ... } =head2 How do I compute the difference of two arrays? How do I compute the intersection of two arrays? @@ -588,29 +1265,140 @@ each element is unique in a given array: push @{ $count{$element} > 1 ? \@intersection : \@difference }, $element; } -=head2 How do I find the first array element for which a condition is true? +Note that this is the I, that is, all elements in +either A or in B but not in both. Think of it as an xor operation. -You can use this if you care about the index: +=head2 How do I test whether two arrays or hashes are equal? - for ($i=0; $i < @array; $i++) { - if ($array[$i] eq "Waldo") { - $found_index = $i; - last; - } +The following code works for single-level arrays. It uses a stringwise +comparison, and does not distinguish defined versus undefined empty +strings. Modify if you have other needs. + + $are_equal = compare_arrays(\@frogs, \@toads); + + sub compare_arrays { + my ($first, $second) = @_; + no warnings; # silence spurious -w undef complaints + return 0 unless @$first == @$second; + for (my $i = 0; $i < @$first; $i++) { + return 0 if $first->[$i] ne $second->[$i]; + } + return 1; } -Now C<$found_index> has what you want. +For multilevel structures, you may wish to use an approach more +like this one. It uses the CPAN module FreezeThaw: + + use FreezeThaw qw(cmpStr); + @a = @b = ( "this", "that", [ "more", "stuff" ] ); + + printf "a and b contain %s arrays\n", + cmpStr(\@a, \@b) == 0 + ? "the same" + : "different"; + +This approach also works for comparing hashes. Here +we'll demonstrate two different answers: + + use FreezeThaw qw(cmpStr cmpStrHard); + + %a = %b = ( "this" => "that", "extra" => [ "more", "stuff" ] ); + $a{EXTRA} = \%b; + $b{EXTRA} = \%a; + + printf "a and b contain %s hashes\n", + cmpStr(\%a, \%b) == 0 ? "the same" : "different"; + + printf "a and b contain %s hashes\n", + cmpStrHard(\%a, \%b) == 0 ? "the same" : "different"; + + +The first reports that both those the hashes contain the same data, +while the second reports that they do not. Which you prefer is left as +an exercise to the reader. + +=head2 How do I find the first array element for which a condition is true? + +To find the first array element which satisfies a condition, you can +use the first() function in the List::Util module, which comes with +Perl 5.8. This example finds the first element that contains "Perl". + + use List::Util qw(first); + + my $element = first { /Perl/ } @array; + +If you cannot use List::Util, you can make your own loop to do the +same thing. Once you find the element, you stop the loop with last. + + my $found; + foreach my $element ( @array ) + { + if( /Perl/ ) { $found = $element; last } + } + +If you want the array index, you can iterate through the indices +and check the array element at each index until you find one +that satisfies the condition. + + my( $found, $index ) = ( undef, -1 ); + for( $i = 0; $i < @array; $i++ ) + { + if( $array[$i] =~ /Perl/ ) + { + $found = $array[$i]; + $index = $i; + last; + } + } =head2 How do I handle linked lists? In general, you usually don't need a linked list in Perl, since with regular arrays, you can push and pop or shift and unshift at either end, -or you can use splice to add and/or remove arbitrary number of elements -at arbitrary points. +or you can use splice to add and/or remove arbitrary number of elements at +arbitrary points. Both pop and shift are both O(1) operations on Perl's +dynamic arrays. In the absence of shifts and pops, push in general +needs to reallocate on the order every log(N) times, and unshift will +need to copy pointers each time. If you really, really wanted, you could use structures as described in L or L and do just what the algorithm book tells you -to do. +to do. For example, imagine a list node like this: + + $node = { + VALUE => 42, + LINK => undef, + }; + +You could walk the list this way: + + print "List: "; + for ($node = $head; $node; $node = $node->{LINK}) { + print $node->{VALUE}, " "; + } + print "\n"; + +You could add to the list this way: + + my ($head, $tail); + $tail = append($head, 1); # grow a new head + for $value ( 2 .. 10 ) { + $tail = append($tail, $value); + } + + sub append { + my($list, $value) = @_; + my $node = { VALUE => $value }; + if ($list) { + $node->{LINK} = $list->{LINK}; + $list->{LINK} = $node; + } else { + $_[0] = $node; # replace caller's version + } + return $node; + } + +But again, Perl's built-in are virtually always good enough. =head2 How do I handle circular lists? @@ -622,8 +1410,36 @@ lists, or you could just do something like this with an array: =head2 How do I shuffle an array randomly? -Here's a shuffling algorithm which works its way through the list, -randomly picking another element to swap the current element with: +If you either have Perl 5.8.0 or later installed, or if you have +Scalar-List-Utils 1.03 or later installed, you can say: + + use List::Util 'shuffle'; + + @shuffled = shuffle(@list); + +If not, you can use a Fisher-Yates shuffle. + + sub fisher_yates_shuffle { + my $deck = shift; # $deck is a reference to an array + my $i = @$deck; + while ($i--) { + my $j = int rand ($i+1); + @$deck[$i,$j] = @$deck[$j,$i]; + } + } + + # shuffle my mpeg collection + # + my @mpeg =