=head1 VERSION DIFFERENCES
-B<NOTE>: 0.99_03 has significant file format differences from prior versions.
-THere will be a backwards-compatibility layer in 1.00, but that is slated for
-a later 0.99_x release. This version is B<NOT> backwards compatible with any
-other release of DBM::Deep.
-
-B<NOTE>: 0.99_01 and above have significant file format differences from 0.983 and
-before. There will be a backwards-compatibility layer in 1.00, but that is
-slated for a later 0.99_x release. This version is B<NOT> backwards compatible
-with 0.983 and before.
+B<NOTE>: 1.0000 has significant file format differences from prior versions.
+THere is a backwards-compatibility layer at C<utils/upgrade_db.pl>. Files
+created by 1.0000 or higher are B<NOT> compatible with scripts using prior
+versions.
=head1 SETUP
=item * num_txns
-This is the maximum number of transactions that can be running at one time. The
-default is two - the HEAD and one for imports. The minimum is two and the
-maximum is 255. The more transactions, the larger and quicker the datafile grows.
+This is the number of transactions that can be running at one time. The
+default is one - the HEAD. The minimum is one and the maximum is 255. The more
+transactions, the larger and quicker the datafile grows.
See L</TRANSACTIONS> below.
This is the number of entries that can be added before a reindexing. The larger
this number is made, the larger a file gets, but the better performance you will
-have. The default and minimum number this can be is 16. There is no maximum, but
-more than 32 isn't recommended.
+have. The default and minimum number this can be is 16. The maximum is 256, but
+more than 64 isn't recommended.
+
+=item * data_sector_size
+
+This is the size in bytes of a given data sector. Data sectors will chain, so
+a value of any size can be stored. However, chaining is expensive in terms of
+time. Setting this value to something close to the expected common length of
+your scalars will improve your performance. If it is too small, your file will
+have a lot of chaining. If it is too large, your file will have a lot of dead
+space in it.
+
+The default for this is 64 bytes. The minimum value is 32 and the maximum is
+256 bytes.
+
+B<Note:> There are between 6 and 10 bytes taken up in each data sector for
+bookkeeping. (It's 4 + the number of bytes in your L</pack_size>.) This is
+included within the data_sector_size, thus the effective value is 6-10 bytes
+less than what you specified.
=item * pack_size
=item * small
-This uses 2-byte offsets, allowing for a maximum file size of 65K
+This uses 2-byte offsets, allowing for a maximum file size of 65 KB.
=item * medium (default)
-This uses 4-byte offsets, allowing for a maximum file size of 2G.
+This uses 4-byte offsets, allowing for a maximum file size of 4 GB.
=item * large
-This uses 8-byte offsets, allowing for a maximum file size of 16XB (exabytes).
+This uses 8-byte offsets, allowing for a maximum file size of 16 XB
+(exabytes). This can only be enabled if your Perl is compiled for 64-bit.
=back
Enable or disable automatic file locking by passing a boolean value to the
C<locking> parameter when constructing your DBM::Deep object (see L<SETUP>
- above).
+above).
my $db = DBM::Deep->new(
file => "foo.db",
These will cause an infinite loop when importing. There are plans to fix this
in a later release.
-B<Note:> With the addition of transactions, importing is performed within a
-transaction, then immediately committed upon success (and rolled back upon
-failre). As a result, you cannot call C<import()> from within a transaction.
-This restriction will be lifted when subtransactions are added in a future
-release.
-
=head2 Exporting
Calling the C<export()> method on an existing DBM::Deep object will return
encryption. Filtering applies to the base DB level, and all child hashes /
arrays. Filter hooks can be specified when your DBM::Deep object is first
constructed, or by calling the C<set_filter()> method at any time. There are
-four available filter hooks, described below:
+four available filter hooks.
+
+=head2 set_filter()
+
+This method takes two paramters - the filter type and the filter subreference.
+The four types are:
=over
$db->set_filter( "filter_store_value", undef );
-=head2 Real-time Encryption Example
-
-Here is a working example that uses the I<Crypt::Blowfish> module to
-do real-time encryption / decryption of keys & values with DBM::Deep Filters.
-Please visit L<http://search.cpan.org/search?module=Crypt::Blowfish> for more
-on I<Crypt::Blowfish>. You'll also need the I<Crypt::CBC> module.
-
- use DBM::Deep;
- use Crypt::Blowfish;
- use Crypt::CBC;
-
- my $cipher = Crypt::CBC->new({
- 'key' => 'my secret key',
- 'cipher' => 'Blowfish',
- 'iv' => '$KJh#(}q',
- 'regenerate_key' => 0,
- 'padding' => 'space',
- 'prepend_iv' => 0
- });
-
- my $db = DBM::Deep->new(
- file => "foo-encrypt.db",
- filter_store_key => \&my_encrypt,
- filter_store_value => \&my_encrypt,
- filter_fetch_key => \&my_decrypt,
- filter_fetch_value => \&my_decrypt,
- );
-
- $db->{key1} = "value1";
- $db->{key2} = "value2";
- print "key1: " . $db->{key1} . "\n";
- print "key2: " . $db->{key2} . "\n";
-
- undef $db;
- exit;
-
- sub my_encrypt {
- return $cipher->encrypt( $_[0] );
- }
- sub my_decrypt {
- return $cipher->decrypt( $_[0] );
- }
-
-=head2 Real-time Compression Example
-
-Here is a working example that uses the I<Compress::Zlib> module to do real-time
-compression / decompression of keys & values with DBM::Deep Filters.
-Please visit L<http://search.cpan.org/search?module=Compress::Zlib> for
-more on I<Compress::Zlib>.
-
- use DBM::Deep;
- use Compress::Zlib;
-
- my $db = DBM::Deep->new(
- file => "foo-compress.db",
- filter_store_key => \&my_compress,
- filter_store_value => \&my_compress,
- filter_fetch_key => \&my_decompress,
- filter_fetch_value => \&my_decompress,
- );
-
- $db->{key1} = "value1";
- $db->{key2} = "value2";
- print "key1: " . $db->{key1} . "\n";
- print "key2: " . $db->{key2} . "\n";
-
- undef $db;
- exit;
-
- sub my_compress {
- return Compress::Zlib::memGzip( $_[0] ) ;
- }
- sub my_decompress {
- return Compress::Zlib::memGunzip( $_[0] ) ;
- }
+=head2 Examples
-B<Note:> Filtering of keys only applies to hashes. Array "keys" are
-actually numerical index numbers, and are not filtered.
+Please read L<DBM::Deep::Manual/> for examples of filters.
=head1 ERROR HANDLING
=head1 LARGEFILE SUPPORT
If you have a 64-bit system, and your Perl is compiled with both LARGEFILE
-and 64-bit support, you I<may> be able to create databases larger than 2 GB.
+and 64-bit support, you I<may> be able to create databases larger than 4 GB.
DBM::Deep by default uses 32-bit file offset tags, but these can be changed
by specifying the 'pack_size' parameter when constructing the file.
parameters are per-file, meaning you can access 32-bit and 64-bit files, as
you choose.
-B<Note:> We have not personally tested files larger than 2 GB -- all my
-systems have only a 32-bit Perl. However, I have received user reports that
-this does indeed work!
+B<Note:> We have not personally tested files larger than 4 GB -- all our
+systems have only a 32-bit Perl. However, we have received user reports that
+this does indeed work.
=head1 LOW-LEVEL ACCESS
data in this structure (be wary of name collision), which is then accessible from
any child hash or array.
-=head1 CUSTOM DIGEST ALGORITHM
-
-DBM::Deep by default uses the I<Message Digest 5> (MD5) algorithm for hashing
-keys. However you can override this, and use another algorithm (such as SHA-256)
-or even write your own. But please note that DBM::Deep currently expects zero
-collisions, so your algorithm has to be I<perfect>, so to speak. Collision
-detection may be introduced in a later version.
-
-You can specify a custom digest algorithm by passing it into the parameter
-list for new(), passing a reference to a subroutine as the 'digest' parameter,
-and the length of the algorithm's hashes (in bytes) as the 'hash_size'
-parameter. Here is a working example that uses a 256-bit hash from the
-I<Digest::SHA256> module. Please see
-L<http://search.cpan.org/search?module=Digest::SHA256> for more information.
-
- use DBM::Deep;
- use Digest::SHA256;
-
- my $context = Digest::SHA256::new(256);
-
- my $db = DBM::Deep->new(
- filename => "foo-sha.db",
- digest => \&my_digest,
- hash_size => 32,
- );
-
- $db->{key1} = "value1";
- $db->{key2} = "value2";
- print "key1: " . $db->{key1} . "\n";
- print "key2: " . $db->{key2} . "\n";
-
- undef $db;
- exit;
-
- sub my_digest {
- return substr( $context->hash($_[0]), 0, 32 );
- }
-
-B<Note:> Your returned digest strings must be B<EXACTLY> the number
-of bytes you specify in the hash_size parameter (in this case 32).
-
-B<Note:> If you do choose to use a custom digest algorithm, you must set it
-every time you access this file. Otherwise, the default (MD5) will be used.
-
=head1 CIRCULAR REFERENCES
-B<NOTE>: DBM::Deep 0.99_03 has turned off circular references pending
-evaluation of some edge cases. I hope to be able to re-enable circular
-references in a future version after 1.00. This means that circular references
-are B<NO LONGER> available.
-
-DBM::Deep has B<experimental> support for circular references. Meaning you
+DBM::Deep has full support for circular references. Meaning you
can have a nested hash key or array element that points to a parent object.
This relationship is stored in the DB file, and is preserved between sessions.
Here is an example:
print $db->{foo} . "\n"; # prints "bar"
print $db->{circle}->{foo} . "\n"; # prints "bar" again
+This also works as expected with array and hash references. So, the following
+works as expected:
+
+ $db->{foo} = [ 1 .. 3 ];
+ $db->{bar} = $db->{foo};
+
+ push @{$db->{foo}}, 42;
+ is( $db->{bar}[-1], 42 ); # Passes
+
+This, however, does I<not> extend to assignments from one DB file to another.
+So, the following will throw an error:
+
+ my $db1 = DBM::Deep->new( "foo.db" );
+ my $db2 = DBM::Deep->new( "bar.db" );
+
+ $db1->{foo} = [];
+ $db2->{foo} = $db1->{foo}; # dies
+
B<Note>: Passing the object to a function that recursively walks the
object tree (such as I<Data::Dumper> or even the built-in C<optimize()> or
C<export()> methods) will result in an infinite loop. This will be fixed in
-a future release.
+a future release by adding singleton support.
=head1 TRANSACTIONS
-New in 0.99_01 is ACID transactions. Every DBM::Deep object is completely
+As of 1.0000, DBM::Deep hass ACID transactions. Every DBM::Deep object is completely
transaction-ready - it is not an option you have to turn on. You do have to
specify how many transactions may run simultaneously (q.v. L</num_txns>).
Transactions in DBM::Deep are done using a variant of the MVCC method, the
same method used by the InnoDB MySQL engine.
-=head2 Software-Transactional Memory
-
-The addition of transactions to this module provides the basis for STM within
-Perl 5. Contention is resolved using a default last-write-wins. Currently,
-this default cannot be changed, but it will be addressed in a future version.
+=head1 MIGRATION
-=head1 PERFORMANCE
+As of 1.0000, the file format has changed. Furthermore, DBM::Deep is now
+designed to potentially change file format between point-releases, if needed to
+support a requested feature. To aid in this, a migration script is provided
+within the CPAN distribution called C<utils/upgrade_db.pl>.
-Because DBM::Deep is a conncurrent datastore, every change is flushed to disk
-immediately and every read goes to disk. This means that DBM::Deep functions
-at the speed of disk (generally 10-20ms) vs. the speed of RAM (generally
-50-70ns), or at least 150-200x slower than the comparable in-memory
-datastructure in Perl.
-
-There are several techniques you can use to speed up how DBM::Deep functions.
-
-=over 4
-
-=item * Put it on a ramdisk
-
-The easiest and quickest mechanism to making DBM::Deep run faster is to create
-a ramdisk and locate the DBM::Deep file there. Doing this as an option may
-become a feature of DBM::Deep, assuming there is a good ramdisk wrapper on CPAN.
-
-=item * Work at the tightest level possible
-
-It is much faster to assign the level of your db that you are working with to
-an intermediate variable than to re-look it up every time. Thus
-
- # BAD
- while ( my ($k, $v) = each %{$db->{foo}{bar}{baz}} ) {
- ...
- }
-
- # GOOD
- my $x = $db->{foo}{bar}{baz};
- while ( my ($k, $v) = each %$x ) {
- ...
- }
-
-=item * Make your file as tight as possible
-
-If you know that you are not going to use more than 65K in your database,
-consider using the C<pack_size =E<gt> 'small'> option. This will instruct
-DBM::Deep to use 16bit addresses, meaning that the seek times will be less.
-
-=back
+B<NOTE:> This script is not installed onto your system because it carries a copy
+of every version prior to the current version.
=head1 TODO
=back
+=head2 Data::Dumper and references
+
+As of 1.0003, support for independent Perl datastructures was added (q.v. L</CIRCULAR REFERENCES>
+for more info). However, because DBM::Deep doesn't properly provide the same
+in-memory data-structure for a given location on disk, Data::Dumper (and
+friends) doesn't properly note this. This will be addressed in a future release.
+
=head2 File corruption
The current level of error handling in DBM::Deep is minimal. Files I<are> checked
B<Devel::Cover> is used to test the code coverage of the tests. Below is the
B<Devel::Cover> report on this distribution's test suite.
- ---------------------------- ------ ------ ------ ------ ------ ------ ------
- File stmt bran cond sub pod time total
- ---------------------------- ------ ------ ------ ------ ------ ------ ------
- blib/lib/DBM/Deep.pm 96.8 87.9 90.5 100.0 89.5 4.5 95.2
- blib/lib/DBM/Deep/Array.pm 100.0 94.3 100.0 100.0 100.0 4.9 98.7
- blib/lib/DBM/Deep/Engine.pm 96.9 85.2 79.7 100.0 0.0 58.2 90.3
- blib/lib/DBM/Deep/File.pm 99.0 88.9 77.8 100.0 0.0 30.0 90.3
- blib/lib/DBM/Deep/Hash.pm 100.0 100.0 100.0 100.0 100.0 2.4 100.0
- Total 97.6 87.9 84.0 100.0 32.1 100.0 92.8
- ---------------------------- ------ ------ ------ ------ ------ ------ ------
+ ----------------------------------- ------ ------ ------ ------ ------ ------
+ File stmt bran cond sub time total
+ ----------------------------------- ------ ------ ------ ------ ------ ------
+ blib/lib/DBM/Deep.pm 94.4 85.0 90.5 100.0 5.0 93.4
+ blib/lib/DBM/Deep/Array.pm 100.0 94.6 100.0 100.0 4.7 98.8
+ blib/lib/DBM/Deep/Engine.pm 97.2 85.8 82.4 100.0 51.3 93.8
+ blib/lib/DBM/Deep/File.pm 97.2 81.6 66.7 100.0 36.5 91.9
+ blib/lib/DBM/Deep/Hash.pm 100.0 100.0 100.0 100.0 2.5 100.0
+ Total 97.2 87.4 83.9 100.0 100.0 94.6
+ ----------------------------------- ------ ------ ------ ------ ------ ------
=head1 MORE INFORMATION
The source code repository is at L<http://svn.perl.org/modules/DBM-Deep>
-=head1 MAINTAINER(S)
+=head1 MAINTAINERS
Rob Kinyon, L<rkinyon@cpan.org>
Originally written by Joseph Huckaby, L<jhuckaby@cpan.org>
+=head1 SPONSORS
+
+Stonehenge Consulting (L<http://www.stonehenge.com/>) sponsored the
+developement of transactions and freespace management, leading to the 1.0000
+release. A great debt of gratitude goes out to them for their continuing
+leadership in and support of the Perl community.
+
=head1 CONTRIBUTORS
The following have contributed greatly to make DBM::Deep what it is today:
=over 4
-=item * Adam Sah and Rich Gaushell
-
-=item * Stonehenge for sponsoring the 1.00 release
+=item * Adam Sah and Rich Gaushell for innumerable contributions early on.
=item * Dan Golden and others at YAPC::NA 2006 for helping me design through transactions.
=head1 LICENSE
Copyright (c) 2007 Rob Kinyon. All Rights Reserved.
-This is free software, you may use it and distribute it under the
-same terms as Perl itself.
+This is free software, you may use it and distribute it under the same terms
+as Perl itself.
=cut
projects / dbsrgits/DBM-Deep.git / blobdiff