3 perldsc - Perl Data Structures Cookbook
7 The single feature most sorely lacking in the Perl programming language
8 prior to its 5.0 release was complex data structures. Even without direct
9 language support, some valiant programmers did manage to emulate them, but
10 it was hard work and not for the faint of heart. You could occasionally
11 get away with the C<$m{$AoA,$b}> notation borrowed from B<awk> in which the
12 keys are actually more like a single concatenated string C<"$AoA$b">, but
13 traversal and sorting were difficult. More desperate programmers even
14 hacked Perl's internal symbol table directly, a strategy that proved hard
15 to develop and maintain--to put it mildly.
17 The 5.0 release of Perl let us have complex data structures. You
18 may now write something like this and all of a sudden, you'd have a array
19 with three dimensions!
30 Alas, however simple this may appear, underneath it's a much more
31 elaborate construct than meets the eye!
33 How do you print it out? Why can't you say just C<print @AoA>? How do
34 you sort it? How can you pass it to a function or get one of these back
35 from a function? Is is an object? Can you save it to disk to read
36 back later? How do you access whole rows or columns of that matrix? Do
37 all the values have to be numeric?
39 As you see, it's quite easy to become confused. While some small portion
40 of the blame for this can be attributed to the reference-based
41 implementation, it's really more due to a lack of existing documentation with
42 examples designed for the beginner.
44 This document is meant to be a detailed but understandable treatment of the
45 many different sorts of data structures you might want to develop. It
46 should also serve as a cookbook of examples. That way, when you need to
47 create one of these complex data structures, you can just pinch, pilfer, or
48 purloin a drop-in example from here.
50 Let's look at each of these possible constructs in detail. There are separate
51 sections on each of the following:
55 =item * arrays of arrays
57 =item * hashes of arrays
59 =item * arrays of hashes
61 =item * hashes of hashes
63 =item * more elaborate constructs
67 But for now, let's look at general issues common to all
68 these types of data structures.
72 The most important thing to understand about all data structures in Perl
73 -- including multidimensional arrays--is that even though they might
74 appear otherwise, Perl C<@ARRAY>s and C<%HASH>es are all internally
75 one-dimensional. They can hold only scalar values (meaning a string,
76 number, or a reference). They cannot directly contain other arrays or
77 hashes, but instead contain I<references> to other arrays or hashes.
79 You can't use a reference to a array or hash in quite the same way that you
80 would a real array or hash. For C or C++ programmers unused to
81 distinguishing between arrays and pointers to the same, this can be
82 confusing. If so, just think of it as the difference between a structure
83 and a pointer to a structure.
85 You can (and should) read more about references in the perlref(1) man
86 page. Briefly, references are rather like pointers that know what they
87 point to. (Objects are also a kind of reference, but we won't be needing
88 them right away--if ever.) This means that when you have something which
89 looks to you like an access to a two-or-more-dimensional array and/or hash,
90 what's really going on is that the base type is
91 merely a one-dimensional entity that contains references to the next
92 level. It's just that you can I<use> it as though it were a
93 two-dimensional one. This is actually the way almost all C
94 multidimensional arrays work as well.
96 $array[7][12] # array of arrays
97 $array[7]{string} # array of hashes
98 $hash{string}[7] # hash of arrays
99 $hash{string}{'another string'} # hash of hashes
101 Now, because the top level contains only references, if you try to print
102 out your array in with a simple print() function, you'll get something
103 that doesn't look very nice, like this:
105 @AoA = ( [2, 3], [4, 5, 7], [0] );
109 ARRAY(0x83c38)ARRAY(0x8b194)ARRAY(0x8b1d0)
112 That's because Perl doesn't (ever) implicitly dereference your variables.
113 If you want to get at the thing a reference is referring to, then you have
114 to do this yourself using either prefix typing indicators, like
115 C<${$blah}>, C<@{$blah}>, C<@{$blah[$i]}>, or else postfix pointer arrows,
116 like C<$a-E<gt>[3]>, C<$h-E<gt>{fred}>, or even C<$ob-E<gt>method()-E<gt>[3]>.
118 =head1 COMMON MISTAKES
120 The two most common mistakes made in constructing something like
121 an array of arrays is either accidentally counting the number of
122 elements or else taking a reference to the same memory location
123 repeatedly. Here's the case where you just get the count instead
127 @array = somefunc($i);
128 $AoA[$i] = @array; # WRONG!
131 That's just the simple case of assigning an array to a scalar and getting
132 its element count. If that's what you really and truly want, then you
133 might do well to consider being a tad more explicit about it, like this:
136 @array = somefunc($i);
137 $counts[$i] = scalar @array;
140 Here's the case of taking a reference to the same memory location
144 @array = somefunc($i);
145 $AoA[$i] = \@array; # WRONG!
148 So, what's the big problem with that? It looks right, doesn't it?
149 After all, I just told you that you need an array of references, so by
150 golly, you've made me one!
152 Unfortunately, while this is true, it's still broken. All the references
153 in @AoA refer to the I<very same place>, and they will therefore all hold
154 whatever was last in @array! It's similar to the problem demonstrated in
155 the following C program:
159 struct passwd *getpwnam(), *rp, *dp;
160 rp = getpwnam("root");
161 dp = getpwnam("daemon");
163 printf("daemon name is %s\nroot name is %s\n",
164 dp->pw_name, rp->pw_name);
169 daemon name is daemon
172 The problem is that both C<rp> and C<dp> are pointers to the same location
173 in memory! In C, you'd have to remember to malloc() yourself some new
174 memory. In Perl, you'll want to use the array constructor C<[]> or the
175 hash constructor C<{}> instead. Here's the right way to do the preceding
176 broken code fragments:
179 @array = somefunc($i);
180 $AoA[$i] = [ @array ];
183 The square brackets make a reference to a new array with a I<copy>
184 of what's in @array at the time of the assignment. This is what
187 Note that this will produce something similar, but it's
192 @{$AoA[$i]} = @array;
195 Is it the same? Well, maybe so--and maybe not. The subtle difference
196 is that when you assign something in square brackets, you know for sure
197 it's always a brand new reference with a new I<copy> of the data.
198 Something else could be going on in this new case with the C<@{$AoA[$i]}}>
199 dereference on the left-hand-side of the assignment. It all depends on
200 whether C<$AoA[$i]> had been undefined to start with, or whether it
201 already contained a reference. If you had already populated @AoA with
204 $AoA[3] = \@another_array;
206 Then the assignment with the indirection on the left-hand-side would
207 use the existing reference that was already there:
211 Of course, this I<would> have the "interesting" effect of clobbering
212 @another_array. (Have you ever noticed how when a programmer says
213 something is "interesting", that rather than meaning "intriguing",
214 they're disturbingly more apt to mean that it's "annoying",
215 "difficult", or both? :-)
217 So just remember always to use the array or hash constructors with C<[]>
218 or C<{}>, and you'll be fine, although it's not always optimally
221 Surprisingly, the following dangerous-looking construct will
222 actually work out fine:
225 my @array = somefunc($i);
229 That's because my() is more of a run-time statement than it is a
230 compile-time declaration I<per se>. This means that the my() variable is
231 remade afresh each time through the loop. So even though it I<looks> as
232 though you stored the same variable reference each time, you actually did
233 not! This is a subtle distinction that can produce more efficient code at
234 the risk of misleading all but the most experienced of programmers. So I
235 usually advise against teaching it to beginners. In fact, except for
236 passing arguments to functions, I seldom like to see the gimme-a-reference
237 operator (backslash) used much at all in code. Instead, I advise
238 beginners that they (and most of the rest of us) should try to use the
239 much more easily understood constructors C<[]> and C<{}> instead of
240 relying upon lexical (or dynamic) scoping and hidden reference-counting to
241 do the right thing behind the scenes.
245 $AoA[$i] = [ @array ]; # usually best
246 $AoA[$i] = \@array; # perilous; just how my() was that array?
247 @{ $AoA[$i] } = @array; # way too tricky for most programmers
250 =head1 CAVEAT ON PRECEDENCE
252 Speaking of things like C<@{$AoA[$i]}>, the following are actually the
255 $aref->[2][2] # clear
256 $$aref[2][2] # confusing
258 That's because Perl's precedence rules on its five prefix dereferencers
259 (which look like someone swearing: C<$ @ * % &>) make them bind more
260 tightly than the postfix subscripting brackets or braces! This will no
261 doubt come as a great shock to the C or C++ programmer, who is quite
262 accustomed to using C<*a[i]> to mean what's pointed to by the I<i'th>
263 element of C<a>. That is, they first take the subscript, and only then
264 dereference the thing at that subscript. That's fine in C, but this isn't C.
266 The seemingly equivalent construct in Perl, C<$$aref[$i]> first does
267 the deref of $aref, making it take $aref as a reference to an
268 array, and then dereference that, and finally tell you the I<i'th> value
269 of the array pointed to by $AoA. If you wanted the C notion, you'd have to
270 write C<${$AoA[$i]}> to force the C<$AoA[$i]> to get evaluated first
271 before the leading C<$> dereferencer.
273 =head1 WHY YOU SHOULD ALWAYS C<use strict>
275 If this is starting to sound scarier than it's worth, relax. Perl has
276 some features to help you avoid its most common pitfalls. The best
277 way to avoid getting confused is to start every program like this:
282 This way, you'll be forced to declare all your variables with my() and
283 also disallow accidental "symbolic dereferencing". Therefore if you'd done
287 [ "fred", "barney", "pebbles", "bambam", "dino", ],
288 [ "homer", "bart", "marge", "maggie", ],
289 [ "george", "jane", "elroy", "judy", ],
294 The compiler would immediately flag that as an error I<at compile time>,
295 because you were accidentally accessing C<@aref>, an undeclared
296 variable, and it would thereby remind you to write instead:
302 Before version 5.002, the standard Perl debugger didn't do a very nice job of
303 printing out complex data structures. With 5.002 or above, the
304 debugger includes several new features, including command line editing as
305 well as the C<x> command to dump out complex data structures. For
306 example, given the assignment to $AoA above, here's the debugger output:
309 $AoA = ARRAY(0x13b5a0)
329 Presented with little comment (these will get their own manpages someday)
330 here are short code examples illustrating access of various
331 types of data structures.
333 =head1 ARRAYS OF ARRAYS
335 =head2 Declaration of a ARRAY OF ARRAYS
338 [ "fred", "barney" ],
339 [ "george", "jane", "elroy" ],
340 [ "homer", "marge", "bart" ],
343 =head2 Generation of a ARRAY OF ARRAYS
347 push @AoA, [ split ];
352 $AoA[$i] = [ somefunc($i) ];
361 # add to an existing row
362 push @{ $AoA[0] }, "wilma", "betty";
364 =head2 Access and Printing of a ARRAY OF ARRAYS
370 $AoA[1][1] =~ s/(\w)/\u$1/;
372 # print the whole thing with refs
374 print "\t [ @$aref ],\n";
377 # print the whole thing with indices
378 for $i ( 0 .. $#AoA ) {
379 print "\t [ @{$AoA[$i]} ],\n";
382 # print the whole thing one at a time
383 for $i ( 0 .. $#AoA ) {
384 for $j ( 0 .. $#{ $AoA[$i] } ) {
385 print "elt $i $j is $AoA[$i][$j]\n";
389 =head1 HASHES OF ARRAYS
391 =head2 Declaration of a HASH OF ARRAYS
394 flintstones => [ "fred", "barney" ],
395 jetsons => [ "george", "jane", "elroy" ],
396 simpsons => [ "homer", "marge", "bart" ],
399 =head2 Generation of a HASH OF ARRAYS
402 # flintstones: fred barney wilma dino
404 next unless s/^(.*?):\s*//;
405 $HoA{$1} = [ split ];
408 # reading from file; more temps
409 # flintstones: fred barney wilma dino
410 while ( $line = <> ) {
411 ($who, $rest) = split /:\s*/, $line, 2;
412 @fields = split ' ', $rest;
413 $HoA{$who} = [ @fields ];
416 # calling a function that returns a list
417 for $group ( "simpsons", "jetsons", "flintstones" ) {
418 $HoA{$group} = [ get_family($group) ];
421 # likewise, but using temps
422 for $group ( "simpsons", "jetsons", "flintstones" ) {
423 @members = get_family($group);
424 $HoA{$group} = [ @members ];
427 # append new members to an existing family
428 push @{ $HoA{"flintstones"} }, "wilma", "betty";
430 =head2 Access and Printing of a HASH OF ARRAYS
433 $HoA{flintstones}[0] = "Fred";
436 $HoA{simpsons}[1] =~ s/(\w)/\u$1/;
438 # print the whole thing
439 foreach $family ( keys %HoA ) {
440 print "$family: @{ $HoA{$family} }\n"
443 # print the whole thing with indices
444 foreach $family ( keys %HoA ) {
446 foreach $i ( 0 .. $#{ $HoA{$family} } ) {
447 print " $i = $HoA{$family}[$i]";
452 # print the whole thing sorted by number of members
453 foreach $family ( sort { @{$HoA{$b}} <=> @{$HoA{$a}} } keys %HoA ) {
454 print "$family: @{ $HoA{$family} }\n"
457 # print the whole thing sorted by number of members and name
458 foreach $family ( sort {
459 @{$HoA{$b}} <=> @{$HoA{$a}}
464 print "$family: ", join(", ", sort @{ $HoA{$family} }), "\n";
467 =head1 ARRAYS OF HASHES
469 =head2 Declaration of a ARRAY OF HASHES
488 =head2 Generation of a ARRAY OF HASHES
491 # format: LEAD=fred FRIEND=barney
494 for $field ( split ) {
495 ($key, $value) = split /=/, $field;
496 $rec->{$key} = $value;
503 # format: LEAD=fred FRIEND=barney
506 push @AoH, { split /[\s+=]/ };
509 # calling a function that returns a key/value pair list, like
510 # "lead","fred","daughter","pebbles"
511 while ( %fields = getnextpairset() ) {
512 push @AoH, { %fields };
515 # likewise, but using no temp vars
517 push @AoH, { parsepairs($_) };
520 # add key/value to an element
521 $AoH[0]{pet} = "dino";
522 $AoH[2]{pet} = "santa's little helper";
524 =head2 Access and Printing of a ARRAY OF HASHES
527 $AoH[0]{lead} = "fred";
530 $AoH[1]{lead} =~ s/(\w)/\u$1/;
532 # print the whole thing with refs
535 for $role ( keys %$href ) {
536 print "$role=$href->{$role} ";
541 # print the whole thing with indices
542 for $i ( 0 .. $#AoH ) {
544 for $role ( keys %{ $AoH[$i] } ) {
545 print "$role=$AoH[$i]{$role} ";
550 # print the whole thing one at a time
551 for $i ( 0 .. $#AoH ) {
552 for $role ( keys %{ $AoH[$i] } ) {
553 print "elt $i $role is $AoH[$i]{$role}\n";
557 =head1 HASHES OF HASHES
559 =head2 Declaration of a HASH OF HASHES
569 "his boy" => "elroy",
578 =head2 Generation of a HASH OF HASHES
581 # flintstones: lead=fred pal=barney wife=wilma pet=dino
583 next unless s/^(.*?):\s*//;
585 for $field ( split ) {
586 ($key, $value) = split /=/, $field;
587 $HoH{$who}{$key} = $value;
591 # reading from file; more temps
593 next unless s/^(.*?):\s*//;
597 for $field ( split ) {
598 ($key, $value) = split /=/, $field;
599 $rec->{$key} = $value;
603 # calling a function that returns a key,value hash
604 for $group ( "simpsons", "jetsons", "flintstones" ) {
605 $HoH{$group} = { get_family($group) };
608 # likewise, but using temps
609 for $group ( "simpsons", "jetsons", "flintstones" ) {
610 %members = get_family($group);
611 $HoH{$group} = { %members };
614 # append new members to an existing family
620 for $what (keys %new_folks) {
621 $HoH{flintstones}{$what} = $new_folks{$what};
624 =head2 Access and Printing of a HASH OF HASHES
627 $HoH{flintstones}{wife} = "wilma";
630 $HoH{simpsons}{lead} =~ s/(\w)/\u$1/;
632 # print the whole thing
633 foreach $family ( keys %HoH ) {
635 for $role ( keys %{ $HoH{$family} } ) {
636 print "$role=$HoH{$family}{$role} ";
641 # print the whole thing somewhat sorted
642 foreach $family ( sort keys %HoH ) {
644 for $role ( sort keys %{ $HoH{$family} } ) {
645 print "$role=$HoH{$family}{$role} ";
651 # print the whole thing sorted by number of members
652 foreach $family ( sort { keys %{$HoH{$b}} <=> keys %{$HoH{$a}} } keys %HoH ) {
654 for $role ( sort keys %{ $HoH{$family} } ) {
655 print "$role=$HoH{$family}{$role} ";
660 # establish a sort order (rank) for each role
662 for ( qw(lead wife son daughter pal pet) ) { $rank{$_} = ++$i }
664 # now print the whole thing sorted by number of members
665 foreach $family ( sort { keys %{ $HoH{$b} } <=> keys %{ $HoH{$a} } } keys %HoH ) {
667 # and print these according to rank order
668 for $role ( sort { $rank{$a} <=> $rank{$b} } keys %{ $HoH{$family} } ) {
669 print "$role=$HoH{$family}{$role} ";
675 =head1 MORE ELABORATE RECORDS
677 =head2 Declaration of MORE ELABORATE RECORDS
679 Here's a sample showing how to create and use a record whose fields are of
680 many different sorts:
684 SEQUENCE => [ @old_values ],
685 LOOKUP => { %some_table },
686 THATCODE => \&some_function,
687 THISCODE => sub { $_[0] ** $_[1] },
693 print $rec->{SEQUENCE}[0];
694 $last = pop @ { $rec->{SEQUENCE} };
696 print $rec->{LOOKUP}{"key"};
697 ($first_k, $first_v) = each %{ $rec->{LOOKUP} };
699 $answer = $rec->{THATCODE}->($arg);
700 $answer = $rec->{THISCODE}->($arg1, $arg2);
702 # careful of extra block braces on fh ref
703 print { $rec->{HANDLE} } "a string\n";
706 $rec->{HANDLE}->autoflush(1);
707 $rec->{HANDLE}->print(" a string\n");
709 =head2 Declaration of a HASH OF COMPLEX RECORDS
713 series => "flintstones",
714 nights => [ qw(monday thursday friday) ],
716 { name => "fred", role => "lead", age => 36, },
717 { name => "wilma", role => "wife", age => 31, },
718 { name => "pebbles", role => "kid", age => 4, },
724 nights => [ qw(wednesday saturday) ],
726 { name => "george", role => "lead", age => 41, },
727 { name => "jane", role => "wife", age => 39, },
728 { name => "elroy", role => "kid", age => 9, },
733 series => "simpsons",
734 nights => [ qw(monday) ],
736 { name => "homer", role => "lead", age => 34, },
737 { name => "marge", role => "wife", age => 37, },
738 { name => "bart", role => "kid", age => 11, },
743 =head2 Generation of a HASH OF COMPLEX RECORDS
746 # this is most easily done by having the file itself be
747 # in the raw data format as shown above. perl is happy
748 # to parse complex data structures if declared as data, so
749 # sometimes it's easiest to do that
751 # here's a piece by piece build up
753 $rec->{series} = "flintstones";
754 $rec->{nights} = [ find_days() ];
757 # assume this file in field=value syntax
759 %fields = split /[\s=]+/;
760 push @members, { %fields };
762 $rec->{members} = [ @members ];
764 # now remember the whole thing
765 $TV{ $rec->{series} } = $rec;
767 ###########################################################
768 # now, you might want to make interesting extra fields that
769 # include pointers back into the same data structure so if
770 # change one piece, it changes everywhere, like for example
771 # if you wanted a {kids} field that was a reference
772 # to an array of the kids' records without having duplicate
773 # records and thus update problems.
774 ###########################################################
775 foreach $family (keys %TV) {
776 $rec = $TV{$family}; # temp pointer
778 for $person ( @{ $rec->{members} } ) {
779 if ($person->{role} =~ /kid|son|daughter/) {
783 # REMEMBER: $rec and $TV{$family} point to same data!!
784 $rec->{kids} = [ @kids ];
787 # you copied the array, but the array itself contains pointers
788 # to uncopied objects. this means that if you make bart get
791 $TV{simpsons}{kids}[0]{age}++;
793 # then this would also change in
794 print $TV{simpsons}{members}[2]{age};
796 # because $TV{simpsons}{kids}[0] and $TV{simpsons}{members}[2]
797 # both point to the same underlying anonymous hash table
799 # print the whole thing
800 foreach $family ( keys %TV ) {
802 print " is on during @{ $TV{$family}{nights} }\n";
803 print "its members are:\n";
804 for $who ( @{ $TV{$family}{members} } ) {
805 print " $who->{name} ($who->{role}), age $who->{age}\n";
807 print "it turns out that $TV{$family}{lead} has ";
808 print scalar ( @{ $TV{$family}{kids} } ), " kids named ";
809 print join (", ", map { $_->{name} } @{ $TV{$family}{kids} } );
815 You cannot easily tie a multilevel data structure (such as a hash of
816 hashes) to a dbm file. The first problem is that all but GDBM and
817 Berkeley DB have size limitations, but beyond that, you also have problems
818 with how references are to be represented on disk. One experimental
819 module that does partially attempt to address this need is the MLDBM
820 module. Check your nearest CPAN site as described in L<perlmodlib> for
821 source code to MLDBM.
825 perlref(1), perllol(1), perldata(1), perlobj(1)
829 Tom Christiansen <F<tchrist@perl.com>>
832 Wed Oct 23 04:57:50 MET DST 1996