use strict;
use warnings;
-our $VERSION = q(0.99_02);
+our $VERSION = q(0.99_03);
use Fcntl qw( :DEFAULT :flock :seek );
use Digest::MD5 ();
=head2 DB OVER NFS
-Beware of using DB files over NFS. DBM::Deep uses flock(), which works well on local
-filesystems, but will NOT protect you from file corruption over NFS. I've heard
-about setting up your NFS server with a locking daemon, then using lockf() to
-lock your files, but your mileage may vary there as well. From what I
-understand, there is no real way to do it. However, if you need access to the
-underlying filehandle in DBM::Deep for using some other kind of locking scheme like
-lockf(), see the L<LOW-LEVEL ACCESS> section above.
+Beware of using DBM::Deep files over NFS. DBM::Deep uses flock(), which works
+well on local filesystems, but will NOT protect you from file corruption over
+NFS. I've heard about setting up your NFS server with a locking daemon, then
+using lockf() to lock your files, but your mileage may vary there as well.
+From what I understand, there is no real way to do it. However, if you need
+access to the underlying filehandle in DBM::Deep for using some other kind of
+locking scheme like lockf(), see the L<LOW-LEVEL ACCESS> section above.
=head2 COPYING OBJECTS
my $copy = $db->clone();
B<Note>: Since clone() here is cloning the object, not the database location, any
-modifications to either $db or $copy will be visible in both.
+modifications to either $db or $copy will be visible to both.
=head2 LARGE ARRAYS
Notice the memory usage increased by only 56K. Test was performed on a 700mHz
x86 box running Linux RedHat 7.2 & Perl 5.6.1.
-=head1 DB FILE FORMAT
-
-In case you were interested in the underlying DB file format, it is documented
-here in this section. You don't need to know this to use the module, it's just
-included for reference.
-
-=head2 SIGNATURE
-
-DBM::Deep files always start with a 32-bit signature to identify the file type.
-This is at offset 0. The signature is "DPDB" in network byte order. This is
-checked for when the file is opened and an error will be thrown if it's not found.
-
-=head2 TAG
-
-The DBM::Deep file is in a I<tagged format>, meaning each section of the file
-has a standard header containing the type of data, the length of data, and then
-the data itself. The type is a single character (1 byte), the length is a
-32-bit unsigned long in network byte order, and the data is, well, the data.
-Here is how it unfolds:
-
-=head2 MASTER INDEX
-
-Immediately after the 32-bit file signature is the I<Master Index> record.
-This is a standard tag header followed by 1024 bytes (in 32-bit mode) or 2048
-bytes (in 64-bit mode) of data. The type is I<H> for hash or I<A> for array,
-depending on how the DBM::Deep object was constructed.
-
-The index works by looking at a I<MD5 Hash> of the hash key (or array index
-number). The first 8-bit char of the MD5 signature is the offset into the
-index, multipled by 4 in 32-bit mode, or 8 in 64-bit mode. The value of the
-index element is a file offset of the next tag for the key/element in question,
-which is usually a I<Bucket List> tag (see below).
-
-The next tag I<could> be another index, depending on how many keys/elements
-exist. See L<RE-INDEXING> below for details.
-
-=head2 BUCKET LIST
-
-A I<Bucket List> is a collection of 16 MD5 hashes for keys/elements, plus
-file offsets to where the actual data is stored. It starts with a standard
-tag header, with type I<B>, and a data size of 320 bytes in 32-bit mode, or
-384 bytes in 64-bit mode. Each MD5 hash is stored in full (16 bytes), plus
-the 32-bit or 64-bit file offset for the I<Bucket> containing the actual data.
-When the list fills up, a I<Re-Index> operation is performed (See
-L<RE-INDEXING> below).
-
-=head2 BUCKET
-
-A I<Bucket> is a tag containing a key/value pair (in hash mode), or a
-index/value pair (in array mode). It starts with a standard tag header with
-type I<D> for scalar data (string, binary, etc.), or it could be a nested
-hash (type I<H>) or array (type I<A>). The value comes just after the tag
-header. The size reported in the tag header is only for the value, but then,
-just after the value is another size (32-bit unsigned long) and then the plain
-key itself. Since the value is likely to be fetched more often than the plain
-key, I figured it would be I<slightly> faster to store the value first.
-
-If the type is I<H> (hash) or I<A> (array), the value is another I<Master Index>
-record for the nested structure, where the process begins all over again.
-
-=head2 RE-INDEXING
-
-After a I<Bucket List> grows to 16 records, its allocated space in the file is
-exhausted. Then, when another key/element comes in, the list is converted to a
-new index record. However, this index will look at the next char in the MD5
-hash, and arrange new Bucket List pointers accordingly. This process is called
-I<Re-Indexing>. Basically, a new index tag is created at the file EOF, and all
-17 (16 + new one) keys/elements are removed from the old Bucket List and
-inserted into the new index. Several new Bucket Lists are created in the
-process, as a new MD5 char from the key is being examined (it is unlikely that
-the keys will all share the same next char of their MD5s).
-
-Because of the way the I<MD5> algorithm works, it is impossible to tell exactly
-when the Bucket Lists will turn into indexes, but the first round tends to
-happen right around 4,000 keys. You will see a I<slight> decrease in
-performance here, but it picks back up pretty quick (see L<SPEED> above). Then
-it takes B<a lot> more keys to exhaust the next level of Bucket Lists. It's
-right around 900,000 keys. This process can continue nearly indefinitely --
-right up until the point the I<MD5> signatures start colliding with each other,
-and this is B<EXTREMELY> rare -- like winning the lottery 5 times in a row AND
-getting struck by lightning while you are walking to cash in your tickets.
-Theoretically, since I<MD5> hashes are 128-bit values, you I<could> have up to
-340,282,366,921,000,000,000,000,000,000,000,000,000 keys/elements (I believe
-this is 340 unodecillion, but don't quote me).
-
-=head2 STORING
-
-When a new key/element is stored, the key (or index number) is first run through
-I<Digest::MD5> to get a 128-bit signature (example, in hex:
-b05783b0773d894396d475ced9d2f4f6). Then, the I<Master Index> record is checked
-for the first char of the signature (in this case I<b0>). If it does not exist,
-a new I<Bucket List> is created for our key (and the next 15 future keys that
-happen to also have I<b> as their first MD5 char). The entire MD5 is written
-to the I<Bucket List> along with the offset of the new I<Bucket> record (EOF at
-this point, unless we are replacing an existing I<Bucket>), where the actual
-data will be stored.
-
-=head2 FETCHING
-
-Fetching an existing key/element involves getting a I<Digest::MD5> of the key
-(or index number), then walking along the indexes. If there are enough
-keys/elements in this DB level, there might be nested indexes, each linked to
-a particular char of the MD5. Finally, a I<Bucket List> is pointed to, which
-contains up to 16 full MD5 hashes. Each is checked for equality to the key in
-question. If we found a match, the I<Bucket> tag is loaded, where the value and
-plain key are stored.
-
-Fetching the plain key occurs when calling the I<first_key()> and I<next_key()>
-methods. In this process the indexes are walked systematically, and each key
-fetched in increasing MD5 order (which is why it appears random). Once the
-I<Bucket> is found, the value is skipped and the plain key returned instead.
-B<Note:> Do not count on keys being fetched as if the MD5 hashes were
-alphabetically sorted. This only happens on an index-level -- as soon as the
-I<Bucket Lists> are hit, the keys will come out in the order they went in --
-so it's pretty much undefined how the keys will come out -- just like Perl's
-built-in hashes.
-
=head1 CODE COVERAGE
B<Devel::Cover> is used to test the code coverage of the tests. Below is the
or send email to L<DBM-Deep@googlegroups.com>. You can also visit #dbm-deep on
irc.perl.org
+The source code repository is at L<http://svn.perl.org/modules/DBM-Deep>
+
=head1 MAINTAINERS
Rob Kinyon, L<rkinyon@cpan.org>
--- /dev/null
+=head1 NAME
+
+DBM::Deep::Internals
+
+=head1 DESCRIPTION
+
+This is a document describing the internal workings of L<DBM::Deep/>. It is
+not necessary to read this document if you only intend to be a user. This
+document is intended for people who either want a deeper understanding of
+specifics of how L<DBM::Deep/> works or who wish to help program
+L<DBM::Deep/>.
+
+=head1 CLASS LAYOUT
+
+L<DBM::Deep/> is broken up into five classes in three inheritance hierarchies.
+
+=over 4
+
+=item *
+
+L<DBM::Deep/> is the parent of L<DBM::Deep::Array/> and L<DBM::Deep::Hash/>.
+These classes form the immediate interface to the outside world. They are the
+classes that provide the TIE mechanisms as well as the OO methods.
+
+=item *
+
+L<DBM::Deep::Engine/> is the layer that deals with the mechanics of reading
+and writing to the file. This is where the logic of the file layout is
+handled.
+
+=item *
+
+L<DBM::Deep::File/> is the layer that deals with the physical file. As a
+singleton that every other object has a reference to, it also provides a place
+to handle datastructure-wide items, such as transactions.
+
+=back
+
+=head1 FILE LAYOUT
+
+DBM::Deep uses a tagged file layout. Every section has a tag, a size, and then
+the data.
+
+=head2 File header
+
+=over 4
+
+=item * File Signature
+
+The first four bytes are 'DPDB' in network byte order, signifying that this is
+a DBM::Deep file.
+
+=item * File tag/size
+
+This is the tagging of the file header. The file used by versions prior to
+1.00 had a different fifth byte, allowing the difference to the determined.
+
+=item * Version
+
+This is four bytes containing the header version. This lets the header change over time.
+
+=item * Transaction information
+
+The current running transactions are stored here, as is the next transaction
+ID.
+
+=item * Constants
+
+These are the file-wide constants that determine how the file is laid out.
+They can only be set upon file creation.
+
+=back
+
+=head2 Index
+
+The Index parts can be tagged either as Hash, Array, or Index. The latter
+is if there was a reindexing due to a bucketlist growing too large. The others
+are the root index for their respective datatypes. The index consists of a
+tag, a size, and then 256 sections containing file locations. Each section
+corresponds to each value representable in a byte.
+
+The index is used as follows - whenever a hashed key is being looked up, the
+first byte is used to determine which location to go to from the root index.
+Then, if that's also an index, the second byte is used, and so forth until a
+bucketlist is found.
+
+=head2 Bucketlist
+
+This is the part that contains the link to the data section. A bucketlist
+defaults to being 16 buckets long (modifiable by the I<max_buckets>
+parameter used when creating a new file). Each bucket contains an MD5 and a
+location of the appropriate key section.
+
+=head2 Key area
+
+This is the part that handles transactional awareness. There are
+I<max_buckets> sections. Each section contains the location to the data
+section, a transaction ID, and whether that transaction considers this key to
+be deleted or not.
+
+=head2 Data area
+
+This is the part that actual stores the key, value, and class (if
+appropriate). The layout is:
+
+=over 4
+
+=item * tag
+
+=item * length of the value
+
+=item * the actual value
+
+=item * keylength
+
+=item * the actual key
+
+=item * a byte indicating if this value has a classname
+
+=item * the classname (if one is there)
+
+=back
+
+The key is stored after the value because the value is requested more often
+than the key.
+
+=head1 PERFORMANCE
+
+L<DBM::Deep/> is written completely in Perl. It also is a multi-process DBM
+that uses the datafile as a method of synchronizing between multiple
+processes. This is unlike most RDBMSes like MySQL and Oracle. Furthermore,
+unlike all RDBMSes, L<DBM::Deep/> stores both the data and the structure of
+that data as it would appear in a Perl program.
+
+=head2 CPU
+
+DBM::Deep attempts to be CPU-light. As it stores all the data on disk,
+DBM::Deep is I/O-bound, not CPU-bound.
+
+=head2 RAM
+
+DBM::Deep uses extremely little RAM relative to the amount of data you can
+access. You can iterate through a million keys (using C<each()>) without
+increasing your memeory usage at all.
+
+=head2 DISK
+
+DBM::Deep is I/O-bound, pure and simple. The faster your disk, the faster
+DBM::Deep will be. Currently, when performing C<my $x = $db-E<gt>{foo}>, there
+are a minimum of 4 seeks and 1332 + N bytes read (where N is the length of your
+data). (All values assume a medium filesize.) The actions take are:
+
+=over 4
+
+=item 1 Lock the file
+
+=item 1 Perform a stat() to determine if the inode has changed
+
+=item 1 Go to the primary index for the $db (1 seek)
+
+=item 1 Read the tag/size of the primary index (5 bytes)
+
+=item 1 Read the body of the primary index (1024 bytes)
+
+=item 1 Go to the bucketlist for this MD5 (1 seek)
+
+=item 1 Read the tag/size of the bucketlist (5 bytes)
+
+=item 1 Read the body of the bucketlist (144 bytes)
+
+=item 1 Go to the keys location for this MD5 (1 seek)
+
+=item 1 Read the tag/size of the keys section (5 bytes)
+
+=item 1 Read the body of the keys location (144 bytes)
+
+=item 1 Go to the data section that corresponds to this transaction ID. (1 seek)
+
+=item 1 Read the tag/size of the data section (5 bytes)
+
+=item 1 Read the value for this data (N bytes)
+
+=item 1 Unlock the file
+
+=back
+
+Every additional level of indexing (if there are enough keys) requires an
+additional seek and the reading of 1029 additional bytes. If the value is
+blessed, an additional 1 seek and 9 + M bytes are read (where M is the length
+of the classname).
+
+Arrays are (currently) even worse because they're considered "funny hashes"
+with the length stored as just another key. This means that if you do any sort
+of lookup with a negative index, this entire process is performed twice - once
+for the length and once for the value.
+
+=cut
projects / dbsrgits/DBM-Deep.git / commitdiff