1 # DB_File.pm -- Perl 5 interface to Berkeley DB
3 # written by Paul Marquess (pmarquess@bfsec.bt.co.uk)
4 # last modified 4th Sept 1996
7 package DB_File::HASHINFO ;
12 @DB_File::HASHINFO::ISA = qw(Tie::Hash);
26 bless { 'bsize' => undef,
40 return $self->{$key} if exists $self->{$key} ;
43 croak "${pkg}::FETCH - Unknown element '$key'" ;
53 if ( exists $self->{$key} )
55 $self->{$key} = $value ;
60 croak "${pkg}::STORE - Unknown element '$key'" ;
68 if ( exists $self->{$key} )
70 delete $self->{$key} ;
75 croak "DB_File::HASHINFO::DELETE - Unknown element '$key'" ;
83 exists $self->{$key} ;
91 croak "${pkg} does not define the method ${method}" ;
94 sub DESTROY { undef %{$_[0]} }
95 sub FIRSTKEY { my $self = shift ; $self->NotHere(ref $self, "FIRSTKEY") }
96 sub NEXTKEY { my $self = shift ; $self->NotHere(ref $self, "NEXTKEY") }
97 sub CLEAR { my $self = shift ; $self->NotHere(ref $self, "CLEAR") }
99 package DB_File::RECNOINFO ;
103 @DB_File::RECNOINFO::ISA = qw(DB_File::HASHINFO) ;
109 bless { 'bval' => undef,
110 'cachesize' => undef,
119 package DB_File::BTREEINFO ;
123 @DB_File::BTREEINFO::ISA = qw(DB_File::HASHINFO) ;
129 bless { 'flags' => undef,
130 'cachesize' => undef,
131 'maxkeypage' => undef,
132 'minkeypage' => undef,
144 use vars qw($VERSION @ISA @EXPORT $AUTOLOAD $DB_BTREE $DB_HASH $DB_RECNO) ;
150 #typedef enum { DB_BTREE, DB_HASH, DB_RECNO } DBTYPE;
151 #$DB_BTREE = TIEHASH DB_File::BTREEINFO ;
152 #$DB_HASH = TIEHASH DB_File::HASHINFO ;
153 #$DB_RECNO = TIEHASH DB_File::RECNOINFO ;
155 $DB_BTREE = new DB_File::BTREEINFO ;
156 $DB_HASH = new DB_File::HASHINFO ;
157 $DB_RECNO = new DB_File::RECNOINFO ;
163 @ISA = qw(Tie::Hash Exporter DynaLoader);
165 $DB_BTREE $DB_HASH $DB_RECNO
200 ($constname = $AUTOLOAD) =~ s/.*:://;
201 my $val = constant($constname, @_ ? $_[0] : 0);
203 if ($! =~ /Invalid/) {
204 $AutoLoader::AUTOLOAD = $AUTOLOAD;
205 goto &AutoLoader::AUTOLOAD;
208 my($pack,$file,$line) = caller;
209 croak "Your vendor has not defined DB macro $constname, used at $file line $line.
213 eval "sub $AUTOLOAD { $val }";
218 # import borrowed from IO::File
219 # exports Fcntl constants if available.
222 my $callpkg = caller;
223 Exporter::export $pkg, $callpkg;
226 Exporter::export 'Fcntl', $callpkg;
230 bootstrap DB_File $VERSION;
232 # Preloaded methods go here. Autoload methods go after __END__, and are
233 # processed by the autosplit program.
238 croak "Usage: \$db->get_dup(key [,flag])\n"
239 unless @_ == 2 or @_ == 3 ;
246 my $wantarray = wantarray ;
252 # get the first value associated with the key, $key
253 #$db->seq($key, $value, R_CURSOR()) ;
255 # iterate through the database until either EOF ($status == 0)
256 # or a different key is encountered ($key ne $origkey).
257 for ($status = $db->seq($key, $value, R_CURSOR()) ;
258 $status == 0 and $key eq $origkey ;
259 $status = $db->seq($key, $value, R_NEXT()) ) {
261 # save the value or count number of matches
264 { ++ $values{$value} }
266 { push (@values, $value) }
273 return ($wantarray ? ($flag ? %values : @values) : $counter) ;
284 DB_File - Perl5 access to Berkeley DB
291 [$X =] tie %hash, 'DB_File', [$filename, $flags, $mode, $DB_HASH] ;
292 [$X =] tie %hash, 'DB_File', $filename, $flags, $mode, $DB_BTREE ;
293 [$X =] tie @array, 'DB_File', $filename, $flags, $mode, $DB_RECNO ;
295 $status = $X->del($key [, $flags]) ;
296 $status = $X->put($key, $value [, $flags]) ;
297 $status = $X->get($key, $value [, $flags]) ;
298 $status = $X->seq($key, $value, $flags) ;
299 $status = $X->sync([$flags]) ;
303 $count = $X->get_dup($key) ;
304 @list = $X->get_dup($key) ;
305 %list = $X->get_dup($key, 1) ;
319 B<DB_File> is a module which allows Perl programs to make use of the
320 facilities provided by Berkeley DB. If you intend to use this
321 module you should really have a copy of the Berkeley DB manual pages at
322 hand. The interface defined here mirrors the Berkeley DB interface
325 Berkeley DB is a C library which provides a consistent interface to a
326 number of database formats. B<DB_File> provides an interface to all
327 three of the database types currently supported by Berkeley DB.
335 This database type allows arbitrary key/value pairs to be stored in data
336 files. This is equivalent to the functionality provided by other
337 hashing packages like DBM, NDBM, ODBM, GDBM, and SDBM. Remember though,
338 the files created using DB_HASH are not compatible with any of the
339 other packages mentioned.
341 A default hashing algorithm, which will be adequate for most
342 applications, is built into Berkeley DB. If you do need to use your own
343 hashing algorithm it is possible to write your own in Perl and have
344 B<DB_File> use it instead.
348 The btree format allows arbitrary key/value pairs to be stored in a
349 sorted, balanced binary tree.
351 As with the DB_HASH format, it is possible to provide a user defined
352 Perl routine to perform the comparison of keys. By default, though, the
353 keys are stored in lexical order.
357 DB_RECNO allows both fixed-length and variable-length flat text files
358 to be manipulated using the same key/value pair interface as in DB_HASH
359 and DB_BTREE. In this case the key will consist of a record (line)
364 =head2 How does DB_File interface to Berkeley DB?
366 B<DB_File> allows access to Berkeley DB files using the tie() mechanism
367 in Perl 5 (for full details, see L<perlfunc/tie()>). This facility
368 allows B<DB_File> to access Berkeley DB files using either an
369 associative array (for DB_HASH & DB_BTREE file types) or an ordinary
370 array (for the DB_RECNO file type).
372 In addition to the tie() interface, it is also possible to access most
373 of the functions provided in the Berkeley DB API directly.
374 See L<THE API INTERFACE>.
376 =head2 Opening a Berkeley DB Database File
378 Berkeley DB uses the function dbopen() to open or create a database.
379 Here is the C prototype for dbopen():
382 dbopen (const char * file, int flags, int mode,
383 DBTYPE type, const void * openinfo)
385 The parameter C<type> is an enumeration which specifies which of the 3
386 interface methods (DB_HASH, DB_BTREE or DB_RECNO) is to be used.
387 Depending on which of these is actually chosen, the final parameter,
388 I<openinfo> points to a data structure which allows tailoring of the
389 specific interface method.
391 This interface is handled slightly differently in B<DB_File>. Here is
392 an equivalent call using B<DB_File>:
394 tie %array, 'DB_File', $filename, $flags, $mode, $DB_HASH ;
396 The C<filename>, C<flags> and C<mode> parameters are the direct
397 equivalent of their dbopen() counterparts. The final parameter $DB_HASH
398 performs the function of both the C<type> and C<openinfo> parameters in
401 In the example above $DB_HASH is actually a pre-defined reference to a
402 hash object. B<DB_File> has three of these pre-defined references.
403 Apart from $DB_HASH, there is also $DB_BTREE and $DB_RECNO.
405 The keys allowed in each of these pre-defined references is limited to
406 the names used in the equivalent C structure. So, for example, the
407 $DB_HASH reference will only allow keys called C<bsize>, C<cachesize>,
408 C<ffactor>, C<hash>, C<lorder> and C<nelem>.
410 To change one of these elements, just assign to it like this:
412 $DB_HASH->{'cachesize'} = 10000 ;
414 The three predefined variables $DB_HASH, $DB_BTREE and $DB_RECNO are
415 usually adequate for most applications. If you do need to create extra
416 instances of these objects, constructors are available for each file
419 Here are examples of the constructors and the valid options available
420 for DB_HASH, DB_BTREE and DB_RECNO respectively.
422 $a = new DB_File::HASHINFO ;
430 $b = new DB_File::BTREEINFO ;
440 $c = new DB_File::RECNOINFO ;
449 The values stored in the hashes above are mostly the direct equivalent
450 of their C counterpart. Like their C counterparts, all are set to a
451 default values - that means you don't have to set I<all> of the
452 values when you only want to change one. Here is an example:
454 $a = new DB_File::HASHINFO ;
455 $a->{'cachesize'} = 12345 ;
456 tie %y, 'DB_File', "filename", $flags, 0777, $a ;
458 A few of the values need extra discussion here. When used, the C
459 equivalent of the keys C<hash>, C<compare> and C<prefix> store pointers
460 to C functions. In B<DB_File> these keys are used to store references
461 to Perl subs. Below are templates for each of the subs:
467 # return the hash value for $data
473 my ($key, $key2) = @_ ;
475 # return 0 if $key1 eq $key2
476 # -1 if $key1 lt $key2
477 # 1 if $key1 gt $key2
478 return (-1 , 0 or 1) ;
483 my ($key, $key2) = @_ ;
485 # return number of bytes of $key2 which are
486 # necessary to determine that it is greater than $key1
490 See L<Changing the BTREE sort order> for an example of using the
493 =head2 Default Parameters
495 It is possible to omit some or all of the final 4 parameters in the
496 call to C<tie> and let them take default values. As DB_HASH is the most
497 common file format used, the call:
499 tie %A, "DB_File", "filename" ;
503 tie %A, "DB_File", "filename", O_CREAT|O_RDWR, 0640, $DB_HASH ;
505 It is also possible to omit the filename parameter as well, so the
512 tie %A, "DB_File", undef, O_CREAT|O_RDWR, 0640, $DB_HASH ;
514 See L<In Memory Databases> for a discussion on the use of C<undef>
515 in place of a filename.
517 =head2 In Memory Databases
519 Berkeley DB allows the creation of in-memory databases by using NULL
520 (that is, a C<(char *)0> in C) in place of the filename. B<DB_File>
521 uses C<undef> instead of NULL to provide this functionality.
525 The DB_HASH file format is probably the most commonly used of the three
526 file formats that B<DB_File> supports. It is also very straightforward
529 =head2 A Simple Example.
531 This example shows how to create a database, add key/value pairs to the
532 database, delete keys/value pairs and finally how to enumerate the
533 contents of the database.
538 tie %h, "DB_File", "fruit", O_RDWR|O_CREAT, 0640, $DB_HASH
539 or die "Cannot open file 'fruit': $!\n";
541 # Add a few key/value pairs to the file
542 $h{"apple"} = "red" ;
543 $h{"orange"} = "orange" ;
544 $h{"banana"} = "yellow" ;
545 $h{"tomato"} = "red" ;
547 # Check for existence of a key
548 print "Banana Exists\n\n" if $h{"banana"} ;
550 # Delete a key/value pair.
553 # print the contents of the file
554 while (($k, $v) = each %h)
555 { print "$k -> $v\n" }
567 Note that the like ordinary associative arrays, the order of the keys
568 retrieved is in an apparently random order.
572 The DB_BTREE format is useful when you want to store data in a given
573 order. By default the keys will be stored in lexical order, but as you
574 will see from the example shown in the next section, it is very easy to
575 define your own sorting function.
577 =head2 Changing the BTREE sort order
579 This script shows how to override the default sorting algorithm that
580 BTREE uses. Instead of using the normal lexical ordering, a case
581 insensitive compare function will be used.
588 my ($key1, $key2) = @_ ;
589 "\L$key1" cmp "\L$key2" ;
592 # specify the Perl sub that will do the comparison
593 $DB_BTREE->{'compare'} = \&Compare ;
595 tie %h, "DB_File", "tree", O_RDWR|O_CREAT, 0640, $DB_BTREE
596 or die "Cannot open file 'tree': $!\n" ;
598 # Add a key/value pair to the file
599 $h{'Wall'} = 'Larry' ;
600 $h{'Smith'} = 'John' ;
601 $h{'mouse'} = 'mickey' ;
602 $h{'duck'} = 'donald' ;
607 # Cycle through the keys printing them in order.
608 # Note it is not necessary to sort the keys as
609 # the btree will have kept them in order automatically.
615 Here is the output from the code above.
621 There are a few point to bear in mind if you want to change the
622 ordering in a BTREE database:
628 The new compare function must be specified when you create the database.
632 You cannot change the ordering once the database has been created. Thus
633 you must use the same compare function every time you access the
638 =head2 Handling duplicate keys
640 The BTREE file type optionally allows a single key to be associated
641 with an arbitrary number of values. This option is enabled by setting
642 the flags element of C<$DB_BTREE> to R_DUP when creating the database.
644 There are some difficulties in using the tied hash interface if you
645 want to manipulate a BTREE database with duplicate keys. Consider this
654 # Enable duplicate records
655 $DB_BTREE->{'flags'} = R_DUP ;
657 tie %h, "DB_File", $filename, O_RDWR|O_CREAT, 0640, $DB_BTREE
658 or die "Cannot open $filename: $!\n";
660 # Add some key/value pairs to the file
661 $h{'Wall'} = 'Larry' ;
662 $h{'Wall'} = 'Brick' ; # Note the duplicate key
663 $h{'Wall'} = 'Brick' ; # Note the duplicate key and value
664 $h{'Smith'} = 'John' ;
665 $h{'mouse'} = 'mickey' ;
667 # iterate through the associative array
668 # and print each key/value pair.
670 { print "$_ -> $h{$_}\n" }
682 As you can see 3 records have been successfully created with key C<Wall>
683 - the only thing is, when they are retrieved from the database they
684 I<seem> to have the same value, namely C<Larry>. The problem is caused
685 by the way that the associative array interface works. Basically, when
686 the associative array interface is used to fetch the value associated
687 with a given key, it will only ever retrieve the first value.
689 Although it may not be immediately obvious from the code above, the
690 associative array interface can be used to write values with duplicate
691 keys, but it cannot be used to read them back from the database.
693 The way to get around this problem is to use the Berkeley DB API method
694 called C<seq>. This method allows sequential access to key/value
695 pairs. See L<THE API INTERFACE> for details of both the C<seq> method
696 and the API in general.
698 Here is the script above rewritten using the C<seq> API method.
706 # Enable duplicate records
707 $DB_BTREE->{'flags'} = R_DUP ;
709 $x = tie %h, "DB_File", $filename, O_RDWR|O_CREAT, 0640, $DB_BTREE
710 or die "Cannot open $filename: $!\n";
712 # Add some key/value pairs to the file
713 $h{'Wall'} = 'Larry' ;
714 $h{'Wall'} = 'Brick' ; # Note the duplicate key
715 $h{'Wall'} = 'Brick' ; # Note the duplicate key and value
716 $h{'Smith'} = 'John' ;
717 $h{'mouse'} = 'mickey' ;
719 # iterate through the btree using seq
720 # and print each key/value pair.
721 for ($status = $x->seq($key, $value, R_FIRST) ;
723 $status = $x->seq($key, $value, R_NEXT) )
724 { print "$key -> $value\n" }
737 This time we have got all the key/value pairs, including the multiple
738 values associated with the key C<Wall>.
740 =head2 The get_dup method.
742 B<DB_File> comes with a utility method, called C<get_dup>, to assist in
743 reading duplicate values from BTREE databases. The method can take the
746 $count = $x->get_dup($key) ;
747 @list = $x->get_dup($key) ;
748 %list = $x->get_dup($key, 1) ;
750 In a scalar context the method returns the number of values associated
751 with the key, C<$key>.
753 In list context, it returns all the values which match C<$key>. Note
754 that the values will be returned in an apparently random order.
756 In list context, if the second parameter is present and evaluates TRUE,
757 the method returns an associative array. The keys of the associative
758 array correspond to the the values that matched in the BTREE and the
759 values of the array are a count of the number of times that particular
760 value occurred in the BTREE.
762 So assuming the database created above, we can use C<get_dup> like
765 $cnt = $x->get_dup("Wall") ;
766 print "Wall occurred $cnt times\n" ;
768 %hash = $x->get_dup("Wall", 1) ;
769 print "Larry is there\n" if $hash{'Larry'} ;
770 print "There are $hash{'Brick'} Brick Walls\n" ;
772 @list = $x->get_dup("Wall") ;
773 print "Wall => [@list]\n" ;
775 @list = $x->get_dup("Smith") ;
776 print "Smith => [@list]\n" ;
778 @list = $x->get_dup("Dog") ;
779 print "Dog => [@list]\n" ;
784 Wall occurred 3 times
786 There are 2 Brick Walls
787 Wall => [Brick Brick Larry]
791 =head2 Matching Partial Keys
793 The BTREE interface has a feature which allows partial keys to be
794 matched. This functionality is I<only> available when the C<seq> method
795 is used along with the R_CURSOR flag.
797 $x->seq($key, $value, R_CURSOR) ;
799 Here is the relevant quote from the dbopen man page where it defines
800 the use of the R_CURSOR flag with seq:
803 Note, for the DB_BTREE access method, the returned key is not
804 necessarily an exact match for the specified key. The returned key
805 is the smallest key greater than or equal to the specified key,
806 permitting partial key matches and range searches.
809 In the example script below, the C<match> sub uses this feature to find
810 and print the first matching key/value pair given a partial key.
820 my $orig_key = $key ;
821 $x->seq($key, $value, R_CURSOR) ;
822 print "$orig_key\t-> $key\t-> $value\n" ;
828 $x = tie %h, "DB_File", $filename, O_RDWR|O_CREAT, 0640, $DB_BTREE
829 or die "Cannot open $filename: $!\n";
831 # Add some key/value pairs to the file
832 $h{'mouse'} = 'mickey' ;
833 $h{'Wall'} = 'Larry' ;
834 $h{'Walls'} = 'Brick' ;
835 $h{'Smith'} = 'John' ;
839 for ($st = $x->seq($key, $value, R_FIRST) ;
841 $st = $x->seq($key, $value, R_NEXT) )
843 { print "$key -> $value\n" }
845 print "\nPARTIAL MATCH\n" ;
869 DB_RECNO provides an interface to flat text files. Both variable and
870 fixed length records are supported.
872 In order to make RECNO more compatible with Perl the array offset for
873 all RECNO arrays begins at 0 rather than 1 as in Berkeley DB.
875 As with normal Perl arrays, a RECNO array can be accessed using
876 negative indexes. The index -1 refers to the last element of the array,
877 -2 the second last, and so on. Attempting to access an element before
878 the start of the array will raise a fatal run-time error.
880 =head2 A Simple Example
882 Here is a simple example that uses RECNO.
887 tie @h, "DB_File", "text", O_RDWR|O_CREAT, 0640, $DB_RECNO
888 or die "Cannot open file 'text': $!\n" ;
890 # Add a few key/value pairs to the file
895 # Check for existence of a key
896 print "Element 1 Exists with value $h[1]\n" if $h[1] ;
898 # use a negative index
899 print "The last element is $h[-1]\n" ;
900 print "The 2nd last element is $h[-2]\n" ;
904 Here is the output from the script:
907 Element 1 Exists with value blue
908 The last element is yellow
909 The 2nd last element is blue
913 As you can see from the example above, the tied array interface is
914 quite limited. To make the interface more useful, a number of methods
915 are supplied with B<DB_File> to simulate the standard array operations
916 that are not currently implemented in Perl's tied array interface. All
917 these methods are accessed via the object returned from the tie call.
919 Here are the methods:
923 =item B<$X-E<gt>push(list) ;>
925 Pushes the elements of C<list> to the end of the array.
927 =item B<$value = $X-E<gt>pop ;>
929 Removes and returns the last element of the array.
931 =item B<$X-E<gt>shift>
933 Removes and returns the first element of the array.
935 =item B<$X-E<gt>unshift(list) ;>
937 Pushes the elements of C<list> to the start of the array.
939 =item B<$X-E<gt>length>
941 Returns the number of elements in the array.
945 =head2 Another Example
947 Here is a more complete example that makes use of some of the methods
948 described above. It also makes use of the API interface directly (see
949 L<THE API INTERFACE>).
952 use vars qw(@h $H $file $i) ;
960 $H = tie @h, "DB_File", $file, O_RDWR|O_CREAT, 0640, $DB_RECNO
961 or die "Cannot open file $file: $!\n" ;
963 # first create a text file to play with
971 # Print the records in order.
973 # The length method is needed here because evaluating a tied
974 # array in a scalar context does not return the number of
975 # elements in the array.
977 print "\nORIGINAL\n" ;
978 foreach $i (0 .. $H->length - 1) {
979 print "$i: $h[$i]\n" ;
982 # use the push & pop methods
985 print "\nThe last record was [$a]\n" ;
987 # and the shift & unshift methods
989 $H->unshift("first") ;
990 print "The first record was [$a]\n" ;
992 # Use the API to add a new record after record 2.
994 $H->put($i, "Newbie", R_IAFTER) ;
996 # and a new record before record 1.
998 $H->put($i, "New One", R_IBEFORE) ;
1003 # now print the records in reverse order
1004 print "\nREVERSE\n" ;
1005 for ($i = $H->length - 1 ; $i >= 0 ; -- $i)
1006 { print "$i: $h[$i]\n" }
1008 # same again, but use the API functions instead
1009 print "\nREVERSE again\n" ;
1011 for ($s = $H->seq($k, $v, R_LAST) ;
1013 $s = $H->seq($k, $v, R_PREV))
1014 { print "$k: $v\n" }
1019 and this is what it outputs:
1028 The last record was [four]
1029 The first record was [zero]
1053 Rather than iterating through the array, C<@h> like this:
1057 it is necessary to use either this:
1059 foreach $i (0 .. $H->length - 1)
1063 for ($a = $H->get($k, $v, R_FIRST) ;
1065 $a = $H->get($k, $v, R_NEXT) )
1069 Notice that both times the C<put> method was used the record index was
1070 specified using a variable, C<$i>, rather than the literal value
1071 itself. This is because C<put> will return the record number of the
1072 inserted line via that parameter.
1076 =head1 THE API INTERFACE
1078 As well as accessing Berkeley DB using a tied hash or array, it is also
1079 possible to make direct use of most of the API functions defined in the
1080 Berkeley DB documentation.
1082 To do this you need to store a copy of the object returned from the tie.
1084 $db = tie %hash, "DB_File", "filename" ;
1086 Once you have done that, you can access the Berkeley DB API functions
1087 as B<DB_File> methods directly like this:
1089 $db->put($key, $value, R_NOOVERWRITE) ;
1091 B<Important:> If you have saved a copy of the object returned from
1092 C<tie>, the underlying database file will I<not> be closed until both
1093 the tied variable is untied and all copies of the saved object are
1094 destroyed. See L<The strict untie pragma> for more details.
1097 $db = tie %hash, "DB_File", "filename"
1098 or die "Cannot tie filename: $!" ;
1103 All the functions defined in L<dbopen> are available except for
1104 close() and dbopen() itself. The B<DB_File> method interface to the
1105 supported functions have been implemented to mirror the way Berkeley DB
1106 works whenever possible. In particular note that:
1112 The methods return a status value. All return 0 on success.
1113 All return -1 to signify an error and set C<$!> to the exact
1114 error code. The return code 1 generally (but not always) means that the
1115 key specified did not exist in the database.
1117 Other return codes are defined. See below and in the Berkeley DB
1118 documentation for details. The Berkeley DB documentation should be used
1119 as the definitive source.
1123 Whenever a Berkeley DB function returns data via one of its parameters,
1124 the equivalent B<DB_File> method does exactly the same.
1128 If you are careful, it is possible to mix API calls with the tied
1129 hash/array interface in the same piece of code. Although only a few of
1130 the methods used to implement the tied interface currently make use of
1131 the cursor, you should always assume that the cursor has been changed
1132 any time the tied hash/array interface is used. As an example, this
1133 code will probably not do what you expect:
1135 $X = tie %x, 'DB_File', $filename, O_RDWR|O_CREAT, 0777, $DB_BTREE
1136 or die "Cannot tie $filename: $!" ;
1138 # Get the first key/value pair and set the cursor
1139 $X->seq($key, $value, R_FIRST) ;
1141 # this line will modify the cursor
1142 $count = scalar keys %x ;
1144 # Get the second key/value pair.
1145 # oops, it didn't, it got the last key/value pair!
1146 $X->seq($key, $value, R_NEXT) ;
1148 The code above can be rearranged to get around the problem, like this:
1150 $X = tie %x, 'DB_File', $filename, O_RDWR|O_CREAT, 0777, $DB_BTREE
1151 or die "Cannot tie $filename: $!" ;
1153 # this line will modify the cursor
1154 $count = scalar keys %x ;
1156 # Get the first key/value pair and set the cursor
1157 $X->seq($key, $value, R_FIRST) ;
1159 # Get the second key/value pair.
1161 $X->seq($key, $value, R_NEXT) ;
1165 All the constants defined in L<dbopen> for use in the flags parameters
1166 in the methods defined below are also available. Refer to the Berkeley
1167 DB documentation for the precise meaning of the flags values.
1169 Below is a list of the methods available.
1173 =item B<$status = $X-E<gt>get($key, $value [, $flags]) ;>
1175 Given a key (C<$key>) this method reads the value associated with it
1176 from the database. The value read from the database is returned in the
1177 C<$value> parameter.
1179 If the key does not exist the method returns 1.
1181 No flags are currently defined for this method.
1183 =item B<$status = $X-E<gt>put($key, $value [, $flags]) ;>
1185 Stores the key/value pair in the database.
1187 If you use either the R_IAFTER or R_IBEFORE flags, the C<$key> parameter
1188 will have the record number of the inserted key/value pair set.
1190 Valid flags are R_CURSOR, R_IAFTER, R_IBEFORE, R_NOOVERWRITE and
1193 =item B<$status = $X-E<gt>del($key [, $flags]) ;>
1195 Removes all key/value pairs with key C<$key> from the database.
1197 A return code of 1 means that the requested key was not in the
1200 R_CURSOR is the only valid flag at present.
1202 =item B<$status = $X-E<gt>fd ;>
1204 Returns the file descriptor for the underlying database.
1206 See L<Locking Databases> for an example of how to make use of the
1207 C<fd> method to lock your database.
1209 =item B<$status = $X-E<gt>seq($key, $value, $flags) ;>
1211 This interface allows sequential retrieval from the database. See
1212 L<dbopen> for full details.
1214 Both the C<$key> and C<$value> parameters will be set to the key/value
1215 pair read from the database.
1217 The flags parameter is mandatory. The valid flag values are R_CURSOR,
1218 R_FIRST, R_LAST, R_NEXT and R_PREV.
1220 =item B<$status = $X-E<gt>sync([$flags]) ;>
1222 Flushes any cached buffers to disk.
1224 R_RECNOSYNC is the only valid flag at present.
1228 =head1 HINTS AND TIPS
1230 =head2 The strict untie pragma
1232 If you run Perl version 5.004 or later (actually any version from the
1233 5.003_01 development release on will suffice) and you make use of the
1234 Berkeley DB API, it is is I<very> strongly recommended that you always
1235 include the C<use strict 'untie'> pragma in any of your scripts that
1236 make use of B<DB_File>.
1238 Even if you don't currently make use of the API interface, it is still
1239 a good idea to include the pragma. It won't affect the performance of
1240 your script, but it will prevent problems in the future.
1242 If possible you should try to run with the full strict pragma, but that
1243 is another story. For further details see L<strict> and
1244 L<perldsc/WHY YOU SHOULD ALWAYS C<use strict>>.
1246 To illustrate the importance of including the untie pragma, here is an
1247 example script that fails in an unexpected place because it doesn't use
1253 $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
1254 or die "Cannot tie first time: $!" ;
1260 $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
1261 or die "Cannot tie second time: $!" ;
1265 When run the script will produce this error message:
1267 Cannot tie second time: Invalid argument at bad.file line 12.
1269 Although the error message above refers to the second tie statement in
1270 the script, the source of the problem is really with the untie
1271 statement that precedes it.
1273 To understand why there is a problem at all with the untie statement,
1274 consider what the tie does for a moment.
1276 Whenever the tie is executed, it creates a logical link between a Perl
1277 variable, the associative array C<%x> in this case, and a Berkeley DB
1278 database, C<tst.fil>. The logical link ensures that all operation on
1279 the associative array are automatically mirrored to the database file.
1281 In normal circumstances the untie is enough to break the logical link
1282 and also close the database. In this particular case there is another
1283 logical link, namely the API object returned from the tie and stored in
1284 C<$X>. Whenever the untie is executed in this case, only the link
1285 between the associative array and the database will be broken. The API
1286 object in C<$X> is still valid, so the database will not be closed.
1288 The end result of this is that when the second tie is executed, the
1289 database will be in an inconsistent state (i.e. it is still opened by
1290 the first tie) - thus the second tie will fail.
1292 If the C<use strict 'untie'> pragma is included in the script, like
1297 use strict 'untie' ;
1299 $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
1300 or die "Cannot tie first time: $!" ;
1306 $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
1307 or die "Cannot tie second time: $!" ;
1309 then the error message becomes:
1311 Can't untie: 1 inner references still exist at bad.file line 11.
1313 which pinpoints the real problem. Finally the script can now be
1314 modified to fix the original problem by destroying the API object
1323 $X = tie %x, 'DB_File', 'tst.fil' , O_RDWR|O_CREAT
1326 =head2 Locking Databases
1328 Concurrent access of a read-write database by several parties requires
1329 them all to use some kind of locking. Here's an example of Tom's that
1330 uses the I<fd> method to get the file descriptor, and then a careful
1331 open() to give something Perl will flock() for you. Run this repeatedly
1332 in the background to watch the locks granted in proper order.
1344 my($oldval, $fd, $db, %db, $value, $key);
1346 $key = shift || 'default';
1347 $value = shift || 'magic';
1351 $db = tie(%db, 'DB_File', '/tmp/foo.db', O_CREAT|O_RDWR, 0644)
1352 || die "dbcreat /tmp/foo.db $!";
1354 print "$$: db fd is $fd\n";
1355 open(DB_FH, "+<&=$fd") || die "dup $!";
1358 unless (flock (DB_FH, LOCK_SH | LOCK_NB)) {
1359 print "$$: CONTENTION; can't read during write update!
1360 Waiting for read lock ($!) ....";
1361 unless (flock (DB_FH, LOCK_SH)) { die "flock: $!" }
1363 print "$$: Read lock granted\n";
1365 $oldval = $db{$key};
1366 print "$$: Old value was $oldval\n";
1367 flock(DB_FH, LOCK_UN);
1369 unless (flock (DB_FH, LOCK_EX | LOCK_NB)) {
1370 print "$$: CONTENTION; must have exclusive lock!
1371 Waiting for write lock ($!) ....";
1372 unless (flock (DB_FH, LOCK_EX)) { die "flock: $!" }
1375 print "$$: Write lock granted\n";
1380 flock(DB_FH, LOCK_UN);
1384 print "$$: Updated db to $key=$value\n";
1386 =head2 Sharing databases with C applications
1388 There is no technical reason why a Berkeley DB database cannot be
1389 shared by both a Perl and a C application.
1391 The vast majority of problems that are reported in this area boil down
1392 to the fact that C strings are NULL terminated, whilst Perl strings are
1395 Here is a real example. Netscape 2.0 keeps a record of the locations you
1396 visit along with the time you last visited them in a DB_HASH database.
1397 This is usually stored in the file F<~/.netscape/history.db>. The key
1398 field in the database is the location string and the value field is the
1399 time the location was last visited stored as a 4 byte binary value.
1401 If you haven't already guessed, the location string is stored with a
1402 terminating NULL. This means you need to be careful when accessing the
1405 Here is a snippet of code that is loosely based on Tom Christiansen's
1406 I<ggh> script (available from your nearest CPAN archive in
1407 F<authors/id/TOMC/scripts/nshist.gz>).
1411 use strict 'untie' ;
1413 $dotdir = $ENV{HOME} || $ENV{LOGNAME};
1415 $HISTORY = "$dotdir/.netscape/history.db";
1417 tie %hist_db, 'DB_File', $HISTORY
1418 or die "Cannot open $HISTORY: $!\n" ;;
1420 # Dump the complete database
1421 while ( ($href, $binary_time) = each %hist_db ) {
1423 # remove the terminating NULL
1424 $href =~ s/\x00$// ;
1426 # convert the binary time into a user friendly string
1427 $date = localtime unpack("V", $binary_time);
1428 print "$date $href\n" ;
1431 # check for the existence of a specific key
1432 # remember to add the NULL
1433 if ( $binary_time = $hist_db{"http://mox.perl.com/\x00"} ) {
1434 $date = localtime unpack("V", $binary_time) ;
1435 print "Last visited mox.perl.com on $date\n" ;
1438 print "Never visited mox.perl.com\n"
1444 =head1 COMMON QUESTIONS
1446 =head2 Why is there Perl source in my database?
1448 If you look at the contents of a database file created by DB_File,
1449 there can sometimes be part of a Perl script included in it.
1451 This happens because Berkeley DB uses dynamic memory to allocate
1452 buffers which will subsequently be written to the database file. Being
1453 dynamic, the memory could have been used for anything before DB
1454 malloced it. As Berkeley DB doesn't clear the memory once it has been
1455 allocated, the unused portions will contain random junk. In the case
1456 where a Perl script gets written to the database, the random junk will
1457 correspond to an area of dynamic memory that happened to be used during
1458 the compilation of the script.
1460 Unless you don't like the possibility of there being part of your Perl
1461 scripts embedded in a database file, this is nothing to worry about.
1463 =head2 How do I store complex data structures with DB_File?
1465 Although B<DB_File> cannot do this directly, there is a module which
1466 can layer transparently over B<DB_File> to accomplish this feat.
1468 Check out the MLDBM module, available on CPAN in the directory
1469 F<modules/by-module/MLDBM>.
1471 =head2 What does "Invalid Argument" mean?
1473 You will get this error message when one of the parameters in the
1474 C<tie> call is wrong. Unfortunately there are quite a few parameters to
1475 get wrong, so it can be difficult to figure out which one it is.
1477 Here are a couple of possibilities:
1483 Attempting to reopen a database without closing it. See
1484 L<The strict untie pragma> for an example.
1488 Using the O_WRONLY flag.
1492 =head2 What does "Bareword 'DB_File' not allowed" mean?
1494 You will encounter this particular error message when you have the
1495 C<strict 'subs'> pragma (or the full strict pragma) in your script.
1496 Consider this script:
1501 tie %x, DB_File, "filename" ;
1503 Running it produces the error in question:
1505 Bareword "DB_File" not allowed while "strict subs" in use
1507 To get around the error, place the word C<DB_File> in either single or
1508 double quotes, like this:
1510 tie %x, "DB_File", "filename" ;
1512 Although it might seem like a real pain, it is really worth the effort
1513 of having a C<use strict> in all your scripts.
1525 When B<DB_File> is opening a database file it no longer terminates the
1526 process if I<dbopen> returned an error. This allows file protection
1527 errors to be caught at run time. Thanks to Judith Grass
1528 E<lt>grass@cybercash.comE<gt> for spotting the bug.
1532 Added prototype support for multiple btree compare callbacks.
1536 B<DB_File> has been in use for over a year. To reflect that, the
1537 version number has been incremented to 1.0.
1539 Added complete support for multiple concurrent callbacks.
1541 Using the I<push> method on an empty list didn't work properly. This
1546 Fixed a core dump problem with SunOS.
1548 The return value from TIEHASH wasn't set to NULL when dbopen returned
1553 Merged OS/2 specific code into DB_File.xs
1555 Removed some redundant code in DB_File.xs.
1557 Documentation update.
1559 Allow negative subscripts with RECNO interface.
1561 Changed the default flags from O_RDWR to O_CREAT|O_RDWR.
1563 The example code which showed how to lock a database needed a call to
1564 C<sync> added. Without it the resultant database file was empty.
1566 Added get_dup method.
1570 Documentation update.
1572 B<DB_File> now imports the constants (O_RDWR, O_CREAT etc.) from Fcntl
1575 The standard hash function C<exists> is now supported.
1577 Modified the behavior of get_dup. When it returns an associative
1578 array, the value is the count of the number of matching BTREE values.
1582 Some older versions of Berkeley DB had problems with fixed length
1583 records using the RECNO file format. The newest version at the time of
1584 writing was 1.85 - this seems to have fixed the problems with RECNO.
1586 I am sure there are bugs in the code. If you do find any, or can
1587 suggest any enhancements, I would welcome your comments.
1591 B<DB_File> comes with the standard Perl source distribution. Look in
1592 the directory F<ext/DB_File>.
1594 Berkeley DB is available at your nearest CPAN archive (see
1595 L<perlmod/"CPAN"> for a list) in F<src/misc/db.1.85.tar.gz>, or via the
1596 host F<ftp.cs.berkeley.edu> in F</ucb/4bsd/db.tar.gz>. It is I<not> under
1599 If you are running IRIX, then get Berkeley DB from
1600 F<http://reality.sgi.com/ariel>. It has the patches necessary to
1601 compile properly on IRIX 5.3.
1605 L<perl(1)>, L<dbopen(3)>, L<hash(3)>, L<recno(3)>, L<btree(3)>
1609 The DB_File interface was written by Paul Marquess
1610 E<lt>pmarquess@bfsec.bt.co.ukE<gt>.
1611 Questions about the DB system itself may be addressed to Keith Bostic
1612 E<lt>bostic@cs.berkeley.eduE<gt>.