=head1 VERSION DIFFERENCES
+B<NOTE>: 1.0020 introduces different engines which are backed by different types
+of storage. There is the original storage (called 'File') and a database storage
+(called 'DBI'). q.v. L</PLUGINS> for more information.
+
B<NOTE>: 1.0000 has significant file format differences from prior versions.
THere is a backwards-compatibility layer at C<utils/upgrade_db.pl>. Files
created by 1.0000 or higher are B<NOT> compatible with scripts using prior
versions.
+=head1 PLUGINS
+
+DBM::Deep is a wrapper around different storage engines. These are:
+
+=head2 File
+
+This is the traditional storage engine, storing the data to a custom file
+format. The parameters accepted are:
+
+=over 4
+
+=item * file
+
+Filename of the DB file to link the handle to. You can pass a full absolute
+filesystem path, partial path, or a plain filename if the file is in the
+current working directory. This is a required parameter (though q.v. fh).
+
+=item * fh
+
+If you want, you can pass in the fh instead of the file. This is most useful for
+doing something like:
+
+ my $db = DBM::Deep->new( { fh => \*DATA } );
+
+You are responsible for making sure that the fh has been opened appropriately
+for your needs. If you open it read-only and attempt to write, an exception will
+be thrown. If you open it write-only or append-only, an exception will be thrown
+immediately as DBM::Deep needs to read from the fh.
+
+=item * file_offset
+
+This is the offset within the file that the DBM::Deep db starts. Most of the
+time, you will not need to set this. However, it's there if you want it.
+
+If you pass in fh and do not set this, it will be set appropriately.
+
+=item * locking
+
+Specifies whether locking is to be enabled. DBM::Deep uses Perl's flock()
+function to lock the database in exclusive mode for writes, and shared mode
+for reads. Pass any true value to enable. This affects the base DB handle
+I<and any child hashes or arrays> that use the same DB file. This is an
+optional parameter, and defaults to 1 (enabled). See L</LOCKING> below for
+more.
+
+=back
+
+=head2 DBI
+
+This is a storage engine that stores the data in a relational database. Funnily
+enough, this engine doesn't work with transactions (yet) as InnoDB doesn't do
+what DBM::Deep needs it to do.
+
+The parameters accepted are:
+
+=over 4
+
+=item * dbh
+
+This is a DBH that's already been opened with L<DBI/connect>.
+
+=item * dbi
+
+This is a hashref containing:
+
+=over 4
+
+=item * dsn
+
+=item * username
+
+=item * password
+
+=item * connect_args
+
+=back
+
+Theses correspond to the 4 parameters L<DBI/connect> takes.
+
+=back
+
+B<NOTE>: This has only been tested with MySQL (with disappointing results). I
+plan on extending this to work with SQLite and PostgreSQL in the next release.
+Oracle, Sybase, and other engines will come later.
+
+=head2 Planned engines
+
+There are plans to extend this functionality to (at least) the following:
+
+=over 4
+
+=item * BDB (and other hash engines like memcached)
+
+=item * NoSQL engines (such as Tokyo Cabinet)
+
+=item * DBIx::Class (and other ORMs)
+
+=back
+
=head1 SETUP
Construction can be done OO-style (which is the recommended way), or using
This opens a new database handle, mapped to the file "foo.db". If this
file does not exist, it will automatically be created. DB files are
opened in "r+" (read/write) mode, and the type of object returned is a
-hash, unless otherwise specified (see L<OPTIONS> below).
+hash, unless otherwise specified (see L</OPTIONS> below).
You can pass a number of options to the constructor to specify things like
locking, autoflush, etc. This is done by passing an inline hash (or hashref):
Notice that the filename is now specified I<inside> the hash with
the "file" parameter, as opposed to being the sole argument to the
constructor. This is required if any options are specified.
-See L<OPTIONS> below for the complete list.
+See L</OPTIONS> below for the complete list.
You can also start with an array instead of a hash. For this, you must
specify the C<type> parameter:
Alternately, you can create a DBM::Deep handle by using Perl's built-in
tie() function. The object returned from tie() can be used to call methods,
such as lock() and unlock(). (That object can be retrieved from the tied
-variable at any time using tied() - please see L<perltie/> for more info.
+variable at any time using tied() - please see L<perltie> for more info.
my %hash;
my $db = tie %hash, "DBM::Deep", "foo.db";
my $db = tie @array, "DBM::Deep", "bar.db";
As with the OO constructor, you can replace the DB filename parameter with
-a hash containing one or more options (see L<OPTIONS> just below for the
+a hash containing one or more options (see L</OPTIONS> just below for the
complete list).
tie %hash, "DBM::Deep", {
=over
-=item * file
-
-Filename of the DB file to link the handle to. You can pass a full absolute
-filesystem path, partial path, or a plain filename if the file is in the
-current working directory. This is a required parameter (though q.v. fh).
-
-=item * fh
-
-If you want, you can pass in the fh instead of the file. This is most useful for doing
-something like:
-
- my $db = DBM::Deep->new( { fh => \*DATA } );
-
-You are responsible for making sure that the fh has been opened appropriately for your
-needs. If you open it read-only and attempt to write, an exception will be thrown. If you
-open it write-only or append-only, an exception will be thrown immediately as DBM::Deep
-needs to read from the fh.
-
-=item * file_offset
-
-This is the offset within the file that the DBM::Deep db starts. Most of the time, you will
-not need to set this. However, it's there if you want it.
-
-If you pass in fh and do not set this, it will be set appropriately.
-
=item * type
This parameter specifies what type of object to create, a hash or array. Use
=over 4
-=item * C<DBM::Deep-E<gt>TYPE_HASH>
+=item * C<<DBM::Deep->TYPE_HASH>>
-=item * C<DBM::Deep-E<gt>TYPE_ARRAY>.
+=item * C<<DBM::Deep->TYPE_ARRAY>>
=back
This only takes effect when beginning a new file. This is an optional
-parameter, and defaults to C<DBM::Deep-E<gt>TYPE_HASH>.
-
-=item * locking
-
-Specifies whether locking is to be enabled. DBM::Deep uses Perl's flock()
-function to lock the database in exclusive mode for writes, and shared mode
-for reads. Pass any true value to enable. This affects the base DB handle
-I<and any child hashes or arrays> that use the same DB file. This is an
-optional parameter, and defaults to 1 (enabled). See L<LOCKING> below for
-more.
+parameter, and defaults to C<<DBM::Deep->TYPE_HASH>>.
=item * autoflush
syntax. Because all DBM::Deep objects are I<tied> to hashes or arrays, you can
treat them as such. DBM::Deep will intercept all reads/writes and direct them
to the right place -- the DB file. This has nothing to do with the
-L<TIE CONSTRUCTION> section above. This simply tells you how to use DBM::Deep
+L</TIE CONSTRUCTION> section above. This simply tells you how to use DBM::Deep
using regular hashes and arrays, rather than calling functions like C<get()>
and C<put()> (although those work too). It is entirely up to you how to want
to access your databases.
This causes an infinite loop, because for each iteration, Perl is calling
FETCH() on the $db handle, resulting in a "new" hash for foo every time, so
it effectively keeps returning the first key over and over again. Instead,
-assign a temporary variable to C<$db->{foo}>, then pass that to each().
+assign a temporary variable to C<<$db->{foo}>>, then pass that to each().
=head2 Arrays
As with hashes, you can treat any DBM::Deep object like a normal Perl array
reference. This includes inserting, removing and manipulating elements,
and the C<push()>, C<pop()>, C<shift()>, C<unshift()> and C<splice()> functions.
-The object must have first been created using type C<DBM::Deep-E<gt>TYPE_ARRAY>,
+The object must have first been created using type C<<DBM::Deep->TYPE_ARRAY>>,
or simply be a nested array reference inside a hash. Example:
my $db = DBM::Deep->new(
push @$db, "bar", "baz";
unshift @$db, "bah";
- my $last_elem = pop @$db; # baz
- my $first_elem = shift @$db; # bah
- my $second_elem = $db->[1]; # bar
+ my $last_elem = pop @$db; # baz
+ my $first_elem = shift @$db; # bah
+ my $second_elem = $db->[1]; # bar
my $num_elements = scalar @$db;
$db->clear(); # hashes or arrays
-=item * lock() / unlock()
+=item * lock() / unlock() / lock_exclusive() / lock_shared()
-q.v. Locking.
+q.v. L</LOCKING> for more info.
=item * optimize()
-Recover lost disk space. This is important to do, especially if you use
-transactions.
+This will compress the datafile so that it takes up as little space as possible.
+There is a freespace manager so that when space is freed up, it is used before
+extending the size of the datafile. But, that freespace just sits in the
+datafile unless C<optimize()> is called.
-=item * import() / export()
+=item * import()
-Data going in and out.
+Unlike simple assignment, C<import()> does not tie the right-hand side. Instead,
+a copy of your data is put into the DB. C<import()> takes either an arrayref (if
+your DB is an array) or a hashref (if your DB is a hash). C<import()> will die
+if anything else is passed in.
+
+=item * export()
+
+This returns a complete copy of the data structure at the point you do the export.
+This copy is in RAM, not on disk like the DB is.
=item * begin_work() / commit() / rollback()
These are the transactional functions. L</TRANSACTIONS> for more information.
+=item * supports( $option )
+
+This returns a boolean depending on if this instance of DBM::Dep supports
+that feature. C<$option> can be one of:
+
+=over 4
+
+=item * transactions
+
+=back
+
=back
=head2 Hashes
Fetches the first element in the array, deletes it, then shifts all the
remaining elements over to take up the space. Returns the element value. This
-method is not recommended with large arrays -- see L<LARGE ARRAYS> below for
+method is not recommended with large arrays -- see L</LARGE ARRAYS> below for
details.
my $elem = $db->shift();
Performs exactly like Perl's built-in function of the same name. See L<perldoc
-f splice> for usage -- it is too complicated to document here. This method is
-not recommended with large arrays -- see L<LARGE ARRAYS> below for details.
+not recommended with large arrays -- see L</LARGE ARRAYS> below for details.
=back
=head1 LOCKING
Enable or disable automatic file locking by passing a boolean value to the
-C<locking> parameter when constructing your DBM::Deep object (see L<SETUP>
+C<locking> parameter when constructing your DBM::Deep object (see L</SETUP>
above).
my $db = DBM::Deep->new(
This causes DBM::Deep to C<flock()> the underlying filehandle with exclusive
mode for writes, and shared mode for reads. This is required if you have
multiple processes accessing the same database file, to avoid file corruption.
-Please note that C<flock()> does NOT work for files over NFS. See L<DB OVER
+Please note that C<flock()> does NOT work for files over NFS. See L</DB OVER
NFS> below for more.
=head2 Explicit Locking
You can explicitly lock a database, so it remains locked for multiple
-actions. This is done by calling the C<lock()> method, and passing an
-optional lock mode argument (defaults to exclusive mode). This is particularly
-useful for things like counters, where the current value needs to be fetched,
-then incremented, then stored again.
+actions. This is done by calling the C<lock_exclusive()> method (for when you
+want to write) or the C<lock_shared()> method (for when you want to read).
+This is particularly useful for things like counters, where the current value
+needs to be fetched, then incremented, then stored again.
- $db->lock();
+ $db->lock_exclusive();
my $counter = $db->get("counter");
$counter++;
$db->put("counter", $counter);
# or...
- $db->lock();
+ $db->lock_exclusive();
$db->{counter}++;
$db->unlock();
-You can pass C<lock()> an optional argument, which specifies which mode to use
-(exclusive or shared). Use one of these two constants:
-C<DBM::Deep-E<gt>LOCK_EX> or C<DBM::Deep-E<gt>LOCK_SH>. These are passed
-directly to C<flock()>, and are the same as the constants defined in Perl's
-L<Fcntl/> module.
+=head2 Win32/Cygwin
- $db->lock( $db->LOCK_SH );
- # something here
- $db->unlock();
+Due to Win32 actually enforcing the read-only status of a shared lock, all
+locks on Win32 and cygwin are exclusive. This is because of how autovivification
+currently works. Hopefully, this will go away in a future release.
=head1 IMPORTING/EXPORTING
=head2 Examples
-Please read L<DBM::Deep::Manual/> for examples of filters.
+Please read L<DBM::Deep::Manual> for examples of filters.
=head1 ERROR HANDLING
by specifying the 'pack_size' parameter when constructing the file.
DBM::Deep->new(
- filename => $filename,
+ file => $filename,
pack_size => 'large',
);
instead of 32-bit longs. After setting these values your DB files have a
theoretical maximum size of 16 XB (exabytes).
-You can also use C<pack_size =E<gt> 'small'> in order to use 16-bit file
+You can also use C<<pack_size => 'small'>> in order to use 16-bit file
offsets.
B<Note:> Changing these values will B<NOT> work for existing database files.
=head1 TODO
The following are items that are planned to be added in future releases. These
-are separate from the L<CAVEATS, ISSUES & BUGS> below.
+are separate from the L</CAVEATS, ISSUES & BUGS> below.
=head2 Sub-Transactions
=head2 Caching
-If a user is willing to assert upon opening the file that this process will be
+If a client is willing to assert upon opening the file that this process will be
the only consumer of that datafile, then there are a number of caching
possibilities that can be taken advantage of. This does, however, mean that
DBM::Deep is more vulnerable to losing data due to unflushed changes. It also
single-process. I have no idea how I'd specify this, though. Suggestions are
welcome.
-=head2 Importing using Data::Walker
-
-Right now, importing is done using C<Clone::clone()> to make a complete copy
-in memory, then tying that copy. It would be much better to use
-L<Data::Walker/> to walk the data structure instead, particularly in the case
-of large datastructures.
-
=head2 Different contention resolution mechanisms
Currently, the only contention resolution mechanism is last-write-wins. This
=item * CODE
-L<Data::Dump::Streamer/> provides a mechanism for serializing coderefs,
+L<Data::Dump::Streamer> provides a mechanism for serializing coderefs,
including saving off all closure state. This would allow for DBM::Deep to
store the code for a subroutine. Then, whenever the subroutine is read, the
code could be C<eval()>'ed into being. However, just as for SCALAR and REF,
=head2 External references and transactions
-If you do C<my $x = $db-E<gt>{foo};>, then start a transaction, $x will be
+If you do C<<my $x = $db->{foo};>>, then start a transaction, $x will be
referencing the database from outside the transaction. A fix for this (and other
issues with how external references into the database) is being looked into. This
is the skipped set of tests in t/39_singletons.t and a related issue is the focus
using C<lockf()> to lock your files, but your mileage may vary there as well.
From what I understand, there is no real way to do it. However, if you need
access to the underlying filehandle in DBM::Deep for using some other kind of
-locking scheme like C<lockf()>, see the L<LOW-LEVEL ACCESS> section above.
+locking scheme like C<lockf()>, see the L</LOW-LEVEL ACCESS> section above.
=head2 Copying Objects
my $copy = $db->clone();
-B<Note>: Since clone() here is cloning the object, not the database location, any
-modifications to either $db or $copy will be visible to both.
+B<Note>: Since clone() here is cloning the object, not the database location,
+any modifications to either $db or $copy will be visible to both.
+
+=head2 Stale References
+
+If you take a reference to an array or hash from the database, it is tied
+to the database itself. This means that if the datum in question is subsequentl
+an invalid location and unpredictable things will happen if you try to use
+it.
+
+So a seemingly innocuous piece of code like this:
+
+ my %hash = %{ $db->{some_hash} };
+
+can fail if another process deletes or clobbers C<< $db->{some_hash} >>
+while the data are being extracted, since S<C<%{ ... }>> is not atomic.
+(This actually happened.) The solution is to lock the database before
+reading the data:
+
+ $db->lock_exclusive;
+ my %hash = %{ $db->{some_hash} };
+ $db->unlock;
=head2 Large Arrays
on DBM::Deep, as every element has to be fetched from disk, then stored again in
a different location. This will be addressed in a future version.
+This has been somewhat addressed so that the cost is constant, regardless of
+what is stored at those locations. So, small arrays with huge data structures in
+them are faster. But, large arrays are still large.
+
=head2 Writeonly Files
-If you pass in a filehandle to new(), you may have opened it in either a readonly or
-writeonly mode. STORE will verify that the filehandle is writable. However, there
-doesn't seem to be a good way to determine if a filehandle is readable. And, if the
-filehandle isn't readable, it's not clear what will happen. So, don't do that.
+If you pass in a filehandle to new(), you may have opened it in either a
+readonly or writeonly mode. STORE will verify that the filehandle is writable.
+However, there doesn't seem to be a good way to determine if a filehandle is
+readable. And, if the filehandle isn't readable, it's not clear what will
+happen. So, don't do that.
=head2 Assignments Within Transactions
=head1 CODE COVERAGE
-B<Devel::Cover> is used to test the code coverage of the tests. Below is the
-B<Devel::Cover> report on this distribution's test suite.
-
- ------------------------------------------ ------ ------ ------ ------ ------
- File stmt bran cond sub total
- ------------------------------------------ ------ ------ ------ ------ ------
- blib/lib/DBM/Deep.pm 96.9 88.3 90.5 100.0 95.7
- blib/lib/DBM/Deep/Array.pm 100.0 95.7 100.0 100.0 99.0
- blib/lib/DBM/Deep/Engine.pm 95.5 84.7 81.6 98.4 92.4
- blib/lib/DBM/Deep/File.pm 97.2 81.6 66.7 100.0 91.9
- blib/lib/DBM/Deep/Hash.pm 100.0 100.0 100.0 100.0 100.0
- Total 96.7 87.0 83.3 99.2 94.1
- ------------------------------------------ ------ ------ ------ ------ ------
+L<Devel::Cover> is used to test the code coverage of the tests. Below is the
+L<Devel::Cover> report on this distribution's test suite.
+
+ ---------------------------- ------ ------ ------ ------ ------ ------ ------
+ File stmt bran cond sub pod time total
+ ---------------------------- ------ ------ ------ ------ ------ ------ ------
+ blib/lib/DBM/Deep.pm 100.0 90.0 81.8 100.0 100.0 32.4 98.2
+ blib/lib/DBM/Deep/Array.pm 100.0 94.4 100.0 100.0 100.0 5.2 98.8
+ blib/lib/DBM/Deep/Engine.pm 100.0 92.9 100.0 100.0 100.0 7.5 100.0
+ ...ib/DBM/Deep/Engine/DBI.pm 93.3 71.2 100.0 100.0 100.0 1.5 89.0
+ ...b/DBM/Deep/Engine/File.pm 91.8 77.8 88.9 100.0 100.0 4.9 89.9
+ blib/lib/DBM/Deep/Hash.pm 100.0 100.0 100.0 100.0 100.0 3.9 100.0
+ .../lib/DBM/Deep/Iterator.pm 100.0 n/a n/a 100.0 100.0 0.0 100.0
+ .../DBM/Deep/Iterator/DBI.pm 100.0 100.0 n/a 100.0 100.0 1.4 100.0
+ ...DBM/Deep/Iterator/File.pm 92.5 84.6 n/a 100.0 66.7 0.6 90.0
+ ...erator/File/BucketList.pm 100.0 75.0 n/a 100.0 66.7 0.4 93.8
+ ...ep/Iterator/File/Index.pm 100.0 100.0 n/a 100.0 100.0 0.2 100.0
+ blib/lib/DBM/Deep/Null.pm 87.5 n/a n/a 75.0 n/a 0.0 83.3
+ blib/lib/DBM/Deep/Sector.pm 91.7 n/a n/a 83.3 0.0 6.7 74.4
+ ...ib/DBM/Deep/Sector/DBI.pm 96.8 83.3 n/a 100.0 0.0 1.0 89.8
+ ...p/Sector/DBI/Reference.pm 98.9 86.4 100.0 100.0 0.0 2.2 89.2
+ ...Deep/Sector/DBI/Scalar.pm 100.0 100.0 n/a 100.0 0.0 1.1 92.9
+ ...b/DBM/Deep/Sector/File.pm 96.0 87.5 100.0 92.3 25.0 2.2 91.0
+ ...Sector/File/BucketList.pm 98.2 85.7 83.3 100.0 0.0 3.3 89.4
+ .../Deep/Sector/File/Data.pm 100.0 n/a n/a 100.0 0.0 0.1 90.9
+ ...Deep/Sector/File/Index.pm 100.0 80.0 33.3 100.0 0.0 0.8 83.1
+ .../Deep/Sector/File/Null.pm 100.0 100.0 n/a 100.0 0.0 0.0 91.7
+ .../Sector/File/Reference.pm 100.0 90.0 80.0 100.0 0.0 1.4 91.5
+ ...eep/Sector/File/Scalar.pm 98.3 87.5 n/a 100.0 0.0 0.8 91.5
+ blib/lib/DBM/Deep/Storage.pm 100.0 n/a n/a 100.0 100.0 0.0 100.0
+ ...b/DBM/Deep/Storage/DBI.pm 97.3 70.8 n/a 100.0 38.5 6.7 87.0
+ .../DBM/Deep/Storage/File.pm 96.6 77.1 80.0 95.7 100.0 15.8 91.8
+ Total 99.2 84.8 84.7 99.8 63.3 100.0 97.6
+ ---------------------------- ------ ------ ------ ------ ------ ------ ------
=head1 MORE INFORMATION
or send email to L<DBM-Deep@googlegroups.com>. You can also visit #dbm-deep on
irc.perl.org
-The source code repository is at L<http://svn.perl.org/modules/DBM-Deep>
+The source code repository is at L<http://github.com/robkinyon/dbm-deep>
=head1 MAINTAINERS