r14864@rob-kinyons-computer: rob | 2007-01-18 23:05:43 -0500

[dbsrgits/DBM-Deep.git] / article.pod

=head0 Adding transactions to DBM::Deep

=head1 What is DBM::Deep?

L<DBM::Deep> is a module written completely in Perl that provides a way of
storing Perl datastructures (scalars, hashes, and arrays) on disk instead of
in memory. The datafile produced is able to be ftp'ed from one machine to
another, regardless of OS or Perl version. There are several reasons why
someone would want to do this.

=over 4

=item * Transparent Persistence

This is the ability to save a set of data structures to disk and retrieve them
later without the vast majority of the program even knowing that the data is
persisted. Furthermore, the datastructure is persisted immediately and not at
set marshalling periods.

=item * Huge datastructures

Normally, datastructures are limited by the size of RAM the server has.
L<DBM::Deep> allows for the size a given datastructure to be limited by disk
instead.

=item * IPC

While not a common use, this allows for inter-process communication without
worrying about the specifics of how a given OS handles IPC.

=back

And, with the release of 1.00, there is now a fourth reason -
software-transactional memory, or STM.

=head1 What are transactions?

Originally from the database world, a transaction is a way of isolating the
effects of a given set of actions, then applying them all at once. It's a way
of saying "I'm going to try the following steps, see if I like the result,
then I want everyone else looking at this datastore to see the effects
immediately." The most common example is taken from banking. Let's say that an
application receives a request to have Joe pay Bob five zorkmids.  Without
transactions, the application would take the money from Joe's account, then
add the money to Bob's account. But, what happens if the application crashes
after debiting Joe, but before crediting Bob? The application has made money
disappear. Or, vice versa, if Bob is credited before Joe is debited, the
application has created moneyN<Yes, the US federal government does use
transactions, so this isn't the cause.>.

With a transaction wrapping the money transfer, if the application crashes in
the middle, it's as if the action never happened. So, when the application
recovers from the crash, Joe and Bob still have the same amount of money in
their accounts as they did before and the transaction can restart and Bob can
finally receive his zorkmids.

=head1 Why add them to DBM::Deep?

The ability to have actions occur in either I<atomically> (as in the previous
example) or I<isolation> from the rest of the users of the data is a powerful
one. This allows for a certain amount of safety and predictability in how data
transformations occur. Imagine, for example, that you have a set of
calculations that will update various variables. However, there are some
situations that will cause you to throw away all results and start over with a
different seed. Without transactions, you would have to put everything into
temporary variables, then transfer the values when the calculations were found
to be successful. With STM, you start a transaction and do your thing within
it. If the calculations succeed, you commit. If they fail, you rollback and
try again. If you're thinking that this is very similar to how SVN or CVS
works, you're absolutely correct - SCMs are transactional in the exact same
way.

=head1 How it happened

=head2 The backstory

This project has easily been the single most complex software endeavor I've
ever undertaken. The first step was to figure out exactly how transactions
were going to work. After several spikesN<These are throwaway coding
explorations.>, the best design seemed to be to make the keys mediate any
transactional information. This means that the entire datafile is unaware of
anything to do with transactions, except for the key's data structure.

=head2 Test-driven development

As transactions have so many edge cases that need to be tested for, it was
imperative that I write my tests along with my development. I subscribe to
TDD, or Test-Driven Development, particularly in my CPAN work. This means that
I try to write a test exercising a piece of functionality I<before> writing
the code providing that functionality. While I haven't always succeeded, doing
this meant that I was able to catch a lot of regressions very quickly. You'll
see this very quickly.

=head2 DBM::Deep's file structure

L<DBM::Deep>'s file structure is a record-based structure. The key (or array
index - arrays are just funny hashes internally) is hashed using MD5 and then
stored in a cascade of Index and Bucketlist records. The bucketlist record
stores the actual key string and pointers to where the data records are
stored. The data records themselves are one of Null, Scalar, or Reference.
Null represents an I<undef>, Scalar represents a string (numbers are
stringified for simplicity) and are allocated in 256byte chunks. Reference
represents an array or hash reference and contains a pointer to an Index and
Bucketlist cascade of its own.

=head2 Transactions in the keys

The first pass was to expand the Bucketlist sector to go from a simple key / 
datapointer mapping to a more complex key / transaction / datapointer mapping.
Initially, I interposed a Transaction record that the bucketlist pointed to.
That then contained the transaction / datapointer mapping. This had the
advantage of changing nothing except for adding one new sector type and the
handling for it. This was very quickly merged into the Bucketlist record to
simplify the resulting code.

This first step got me to the point where I could pass the following test:

  my $db1 = DBM::Deep->new( $filename );
  my $db2 = DBM::Deep->new( $filename );

  $db1->{abc} = 'foo';

  is( $db1->{abc}, 'foo' );
  is( $db2->{abc}, 'foo' );

  $db1->begin_work();

      is( $db1->{abc}, 'foo' );
      is( $db2->{abc}, 'foo' );

      $db1->{abc} = 'floober';

      is( $db1->{abc}, 'floober' );
      is( $db2->{abc}, 'foo' );

Just that much was a major accomplishment. The first pass only looked in the
transaction's spot in the bucket for that key. And, that passed my first tests
because I didn't check that C<$db1-E<gt>{abc}> was still 'foo' I<before>
modifying it in the transaction. To pass that test, the code for retrieval
needed to look first in the transaction's spot and if that spot had never been
assigned to, look at the spot for the HEAD.

=head2 The concept of the HEAD

This is a concept borrowed from SVN. In SVN, the HEAD revision is the latest
revision checked into the repository. When you do a ocal modification, you're
doing a modification to the HEAD. Then, you choose to either check in your
code (commit()) or revert (rollback()).

In L<DBM::Deep>, I chose to make the HEAD transaction ID 0. This has several
benefits:

=over 4

=item * Easy identifiaction of a transaction

C<if ( $trans_id ) {}> will run the code iff we are in a running transaction.

=item * The HEAD is the first bucket

In a given bucket, the HEAD is the first datapointer because we mutliply the
size of the transactional bookkeeping by the transaction ID to find the offset
to seek into the file.

=back

=head2 Tracking modified buckets

This gets commit

=head2 Deleted marker

This gets delete within the txn and assignment within the txn

=head2 Freespace management

This keeps the file from exploding in size, but it interacts with the deleted
marker.

=cut