From: rkinyon Date: Fri, 28 Apr 2006 18:46:27 +0000 (+0000) Subject: Broke out a large portion of POD into the Cookbook and Internals POD files X-Git-Tag: 0-99_03~55 X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=commitdiff_plain;h=d8db292980a47f863c228e8968c4069fd45d35d4;p=dbsrgits%2FDBM-Deep.git Broke out a large portion of POD into the Cookbook and Internals POD files --- diff --git a/MANIFEST b/MANIFEST index 4394039..1f879f2 100644 --- a/MANIFEST +++ b/MANIFEST @@ -9,6 +9,8 @@ lib/DBM/Deep/Engine.pm lib/DBM/Deep/File.pm lib/DBM/Deep/Array.pm lib/DBM/Deep/Hash.pm +lib/DBM/Deep/Cookbook.pod +lib/DBM/Deep/Internals.pod t/common.pm t/01_basic.t t/02_hash.t diff --git a/lib/DBM/Deep.pm b/lib/DBM/Deep.pm index 92f3053..908fe01 100644 --- a/lib/DBM/Deep.pm +++ b/lib/DBM/Deep.pm @@ -34,7 +34,7 @@ use 5.6.0; use strict; use warnings; -our $VERSION = q(0.99_02); +our $VERSION = q(0.99_03); use Fcntl qw( :DEFAULT :flock :seek ); use Digest::MD5 (); @@ -1648,13 +1648,13 @@ be addressed in a later version of DBM::Deep. =head2 DB OVER NFS -Beware of using DB files over NFS. DBM::Deep uses flock(), which works well on local -filesystems, but will NOT protect you from file corruption over NFS. I've heard -about setting up your NFS server with a locking daemon, then using lockf() to -lock your files, but your mileage may vary there as well. From what I -understand, there is no real way to do it. However, if you need access to the -underlying filehandle in DBM::Deep for using some other kind of locking scheme like -lockf(), see the L section above. +Beware of using DBM::Deep files over NFS. DBM::Deep uses flock(), which works +well on local filesystems, but will NOT protect you from file corruption over +NFS. I've heard about setting up your NFS server with a locking daemon, then +using lockf() to lock your files, but your mileage may vary there as well. +From what I understand, there is no real way to do it. However, if you need +access to the underlying filehandle in DBM::Deep for using some other kind of +locking scheme like lockf(), see the L section above. =head2 COPYING OBJECTS @@ -1665,7 +1665,7 @@ returns a new, blessed, tied hash or array to the same level in the DB. my $copy = $db->clone(); B: Since clone() here is cloning the object, not the database location, any -modifications to either $db or $copy will be visible in both. +modifications to either $db or $copy will be visible to both. =head2 LARGE ARRAYS @@ -1758,123 +1758,6 @@ process after storing and fetching 1,000,000 keys: Notice the memory usage increased by only 56K. Test was performed on a 700mHz x86 box running Linux RedHat 7.2 & Perl 5.6.1. -=head1 DB FILE FORMAT - -In case you were interested in the underlying DB file format, it is documented -here in this section. You don't need to know this to use the module, it's just -included for reference. - -=head2 SIGNATURE - -DBM::Deep files always start with a 32-bit signature to identify the file type. -This is at offset 0. The signature is "DPDB" in network byte order. This is -checked for when the file is opened and an error will be thrown if it's not found. - -=head2 TAG - -The DBM::Deep file is in a I, meaning each section of the file -has a standard header containing the type of data, the length of data, and then -the data itself. The type is a single character (1 byte), the length is a -32-bit unsigned long in network byte order, and the data is, well, the data. -Here is how it unfolds: - -=head2 MASTER INDEX - -Immediately after the 32-bit file signature is the I record. -This is a standard tag header followed by 1024 bytes (in 32-bit mode) or 2048 -bytes (in 64-bit mode) of data. The type is I for hash or I for array, -depending on how the DBM::Deep object was constructed. - -The index works by looking at a I of the hash key (or array index -number). The first 8-bit char of the MD5 signature is the offset into the -index, multipled by 4 in 32-bit mode, or 8 in 64-bit mode. The value of the -index element is a file offset of the next tag for the key/element in question, -which is usually a I tag (see below). - -The next tag I be another index, depending on how many keys/elements -exist. See L below for details. - -=head2 BUCKET LIST - -A I is a collection of 16 MD5 hashes for keys/elements, plus -file offsets to where the actual data is stored. It starts with a standard -tag header, with type I, and a data size of 320 bytes in 32-bit mode, or -384 bytes in 64-bit mode. Each MD5 hash is stored in full (16 bytes), plus -the 32-bit or 64-bit file offset for the I containing the actual data. -When the list fills up, a I operation is performed (See -L below). - -=head2 BUCKET - -A I is a tag containing a key/value pair (in hash mode), or a -index/value pair (in array mode). It starts with a standard tag header with -type I for scalar data (string, binary, etc.), or it could be a nested -hash (type I) or array (type I). The value comes just after the tag -header. The size reported in the tag header is only for the value, but then, -just after the value is another size (32-bit unsigned long) and then the plain -key itself. Since the value is likely to be fetched more often than the plain -key, I figured it would be I faster to store the value first. - -If the type is I (hash) or I (array), the value is another I -record for the nested structure, where the process begins all over again. - -=head2 RE-INDEXING - -After a I grows to 16 records, its allocated space in the file is -exhausted. Then, when another key/element comes in, the list is converted to a -new index record. However, this index will look at the next char in the MD5 -hash, and arrange new Bucket List pointers accordingly. This process is called -I. Basically, a new index tag is created at the file EOF, and all -17 (16 + new one) keys/elements are removed from the old Bucket List and -inserted into the new index. Several new Bucket Lists are created in the -process, as a new MD5 char from the key is being examined (it is unlikely that -the keys will all share the same next char of their MD5s). - -Because of the way the I algorithm works, it is impossible to tell exactly -when the Bucket Lists will turn into indexes, but the first round tends to -happen right around 4,000 keys. You will see a I decrease in -performance here, but it picks back up pretty quick (see L above). Then -it takes B more keys to exhaust the next level of Bucket Lists. It's -right around 900,000 keys. This process can continue nearly indefinitely -- -right up until the point the I signatures start colliding with each other, -and this is B rare -- like winning the lottery 5 times in a row AND -getting struck by lightning while you are walking to cash in your tickets. -Theoretically, since I hashes are 128-bit values, you I have up to -340,282,366,921,000,000,000,000,000,000,000,000,000 keys/elements (I believe -this is 340 unodecillion, but don't quote me). - -=head2 STORING - -When a new key/element is stored, the key (or index number) is first run through -I to get a 128-bit signature (example, in hex: -b05783b0773d894396d475ced9d2f4f6). Then, the I record is checked -for the first char of the signature (in this case I). If it does not exist, -a new I is created for our key (and the next 15 future keys that -happen to also have I as their first MD5 char). The entire MD5 is written -to the I along with the offset of the new I record (EOF at -this point, unless we are replacing an existing I), where the actual -data will be stored. - -=head2 FETCHING - -Fetching an existing key/element involves getting a I of the key -(or index number), then walking along the indexes. If there are enough -keys/elements in this DB level, there might be nested indexes, each linked to -a particular char of the MD5. Finally, a I is pointed to, which -contains up to 16 full MD5 hashes. Each is checked for equality to the key in -question. If we found a match, the I tag is loaded, where the value and -plain key are stored. - -Fetching the plain key occurs when calling the I and I -methods. In this process the indexes are walked systematically, and each key -fetched in increasing MD5 order (which is why it appears random). Once the -I is found, the value is skipped and the plain key returned instead. -B Do not count on keys being fetched as if the MD5 hashes were -alphabetically sorted. This only happens on an index-level -- as soon as the -I are hit, the keys will come out in the order they went in -- -so it's pretty much undefined how the keys will come out -- just like Perl's -built-in hashes. - =head1 CODE COVERAGE B is used to test the code coverage of the tests. Below is the @@ -1897,6 +1780,8 @@ Check out the DBM::Deep Google Group at L. You can also visit #dbm-deep on irc.perl.org +The source code repository is at L + =head1 MAINTAINERS Rob Kinyon, L diff --git a/lib/DBM/Deep/Cookbook.pod b/lib/DBM/Deep/Cookbook.pod new file mode 100644 index 0000000..61b8163 --- /dev/null +++ b/lib/DBM/Deep/Cookbook.pod @@ -0,0 +1,29 @@ +=head1 NAME + +DBM::Deep::Cookbook + +=head1 DESCRIPTION + +This is the Cookbook for L. It contains useful tips and tricks, +plus some examples of how to do common tasks. + +=head1 RECIPES + +=head2 UTF8 data + +When you're using UTF8 data, you may run into the "Wide character in print" +warning. To fix that in 5.8+, do the following: + + my $db = DBM::Deep->new( ... ); + binmode $db->_fh, ":utf8"; + +In 5.6, you will have to do the following: + + my $db = DBM::Deep->new( ... ); + $db->set_filter( 'store_value' => sub { pack "U0C*", unpack "C*", $_[0] } ); + $db->set_filter( 'retrieve_value' => sub { pack "C*", unpack "U0C*", $_[0] } ); + +In a future version, you will be able to specify C 1> and +L will do these things for you. + +=end diff --git a/lib/DBM/Deep/Internals.pod b/lib/DBM/Deep/Internals.pod new file mode 100644 index 0000000..d80c816 --- /dev/null +++ b/lib/DBM/Deep/Internals.pod @@ -0,0 +1,197 @@ +=head1 NAME + +DBM::Deep::Internals + +=head1 DESCRIPTION + +This is a document describing the internal workings of L. It is +not necessary to read this document if you only intend to be a user. This +document is intended for people who either want a deeper understanding of +specifics of how L works or who wish to help program +L. + +=head1 CLASS LAYOUT + +L is broken up into five classes in three inheritance hierarchies. + +=over 4 + +=item * + +L is the parent of L and L. +These classes form the immediate interface to the outside world. They are the +classes that provide the TIE mechanisms as well as the OO methods. + +=item * + +L is the layer that deals with the mechanics of reading +and writing to the file. This is where the logic of the file layout is +handled. + +=item * + +L is the layer that deals with the physical file. As a +singleton that every other object has a reference to, it also provides a place +to handle datastructure-wide items, such as transactions. + +=back + +=head1 FILE LAYOUT + +DBM::Deep uses a tagged file layout. Every section has a tag, a size, and then +the data. + +=head2 File header + +=over 4 + +=item * File Signature + +The first four bytes are 'DPDB' in network byte order, signifying that this is +a DBM::Deep file. + +=item * File tag/size + +This is the tagging of the file header. The file used by versions prior to +1.00 had a different fifth byte, allowing the difference to the determined. + +=item * Version + +This is four bytes containing the header version. This lets the header change over time. + +=item * Transaction information + +The current running transactions are stored here, as is the next transaction +ID. + +=item * Constants + +These are the file-wide constants that determine how the file is laid out. +They can only be set upon file creation. + +=back + +=head2 Index + +The Index parts can be tagged either as Hash, Array, or Index. The latter +is if there was a reindexing due to a bucketlist growing too large. The others +are the root index for their respective datatypes. The index consists of a +tag, a size, and then 256 sections containing file locations. Each section +corresponds to each value representable in a byte. + +The index is used as follows - whenever a hashed key is being looked up, the +first byte is used to determine which location to go to from the root index. +Then, if that's also an index, the second byte is used, and so forth until a +bucketlist is found. + +=head2 Bucketlist + +This is the part that contains the link to the data section. A bucketlist +defaults to being 16 buckets long (modifiable by the I +parameter used when creating a new file). Each bucket contains an MD5 and a +location of the appropriate key section. + +=head2 Key area + +This is the part that handles transactional awareness. There are +I sections. Each section contains the location to the data +section, a transaction ID, and whether that transaction considers this key to +be deleted or not. + +=head2 Data area + +This is the part that actual stores the key, value, and class (if +appropriate). The layout is: + +=over 4 + +=item * tag + +=item * length of the value + +=item * the actual value + +=item * keylength + +=item * the actual key + +=item * a byte indicating if this value has a classname + +=item * the classname (if one is there) + +=back + +The key is stored after the value because the value is requested more often +than the key. + +=head1 PERFORMANCE + +L is written completely in Perl. It also is a multi-process DBM +that uses the datafile as a method of synchronizing between multiple +processes. This is unlike most RDBMSes like MySQL and Oracle. Furthermore, +unlike all RDBMSes, L stores both the data and the structure of +that data as it would appear in a Perl program. + +=head2 CPU + +DBM::Deep attempts to be CPU-light. As it stores all the data on disk, +DBM::Deep is I/O-bound, not CPU-bound. + +=head2 RAM + +DBM::Deep uses extremely little RAM relative to the amount of data you can +access. You can iterate through a million keys (using C) without +increasing your memeory usage at all. + +=head2 DISK + +DBM::Deep is I/O-bound, pure and simple. The faster your disk, the faster +DBM::Deep will be. Currently, when performing C{foo}>, there +are a minimum of 4 seeks and 1332 + N bytes read (where N is the length of your +data). (All values assume a medium filesize.) The actions take are: + +=over 4 + +=item 1 Lock the file + +=item 1 Perform a stat() to determine if the inode has changed + +=item 1 Go to the primary index for the $db (1 seek) + +=item 1 Read the tag/size of the primary index (5 bytes) + +=item 1 Read the body of the primary index (1024 bytes) + +=item 1 Go to the bucketlist for this MD5 (1 seek) + +=item 1 Read the tag/size of the bucketlist (5 bytes) + +=item 1 Read the body of the bucketlist (144 bytes) + +=item 1 Go to the keys location for this MD5 (1 seek) + +=item 1 Read the tag/size of the keys section (5 bytes) + +=item 1 Read the body of the keys location (144 bytes) + +=item 1 Go to the data section that corresponds to this transaction ID. (1 seek) + +=item 1 Read the tag/size of the data section (5 bytes) + +=item 1 Read the value for this data (N bytes) + +=item 1 Unlock the file + +=back + +Every additional level of indexing (if there are enough keys) requires an +additional seek and the reading of 1029 additional bytes. If the value is +blessed, an additional 1 seek and 9 + M bytes are read (where M is the length +of the classname). + +Arrays are (currently) even worse because they're considered "funny hashes" +with the length stored as just another key. This means that if you do any sort +of lookup with a negative index, this entire process is performed twice - once +for the length and once for the value. + +=cut