=head0 Adding transactions to DBM::Deep
-For the past 9 months, I've been working on adding transactions to
+For the past nine months, I've been working on adding transactions to
L<DBM::Deep|DBM::Deep>. During that time, I looked far and wide for an
accessible description of how a programmer should go about implementing
transactions. The only things I found were either extremely pedantic academic
-papers or code. The former weren't very easy to read and the latter were less
-soN<Have I<you> ever tried to read the source for BDB or InnoDB? Reading
-perl's source is easier.>. This is the article I wished I'd been able to read
-nine months ago.
+papers or the code of complex applications. The former weren't very easy to
+read and the latter were less so N<Have I<you> ever tried to read the source
+for BDB or InnoDB? Reading perl's source is easier.>. This is the article I
+wished I'd been able to read nine months ago.
=head1 What is DBM::Deep?
=item * Transparent Persistence
-This is the ability to save a set of data structures to disk and retrieve them
-later without the vast majority of the program even knowing that the data is
+This is the ability to save a set of datastructures to disk and retrieve them
+later without the vast majority of the program ever knowing that the data is
persisted. Furthermore, the datastructure is persisted immediately and not at
set marshalling periods.
Normally, datastructures are limited by the size of RAM the server has.
L<DBM::Deep|DBM::Deep> allows for the size a given datastructure to be limited by disk
-instead (up to the given perl's largefile support).
+size instead (up to the given perl's largefile support).
=item * Database
-Once you have persistent datastructures, you start wanting to have multiple
-processes be able to use that data. (More on this later.)
+Most programmers hear the word "database" and think "relational database
+management system." A database is a more general term meaning "place one
+stores data." This can be relational, object, or something else. The software
+used to manage and query a database is a "database management system" (DBMS).
+
+L<DBM::Deep|DBM::Deep> provides one half of a DBMS - the data storage part.
+Once the datastructures on disk, L<DBM::Deep|DBM::Deep> provides the
+capability to allow multiple processes to access the data.
=back
L<DBM::Deep|DBM::Deep>'s file structure is record-based. The key (or array
index - arrays are currently just funny hashes internally) is hashed using MD5
-and then stored in a cascade of Index and Bucketlist records. The bucketlist
+and then stored in a cascade of Index and bucketlist records. The bucketlist
record stores the actual key string and pointers to where the data records are
stored. The data records themselves are one of Null, Scalar, or Reference.
Null represents an I<undef>, Scalar represents a string (numbers are
stringified internally for simplicity) and are allocated in 256byte chunks.
Reference represent an array or hash reference and contains a pointer to an
-Index and Bucketlist cascade of its own. Reference will also store the class
+Index and bucketlist cascade of its own. Reference will also store the class
the hash or array reference is blessed into, meaning that almost all objects
can be stored safely.
of saying "I'm going to try the following steps, see if I like the result,
then I want everyone else looking at this datastore to see the results
immediately." The most common example is taken from banking. Let's say that an
-application receives a request to have Joe pay Bob five zorkmids. Without
+application receives a request to have Joe pay Bob five zorkmids. Without
transactions, the application would take the money from Joe's account, then
add the money to Bob's account. But, what happens if the application crashes
after debiting Joe, but before crediting Bob? The application has made money
When the transaction begins and when it is committed, the database must be in
a legal state. This condition doesn't apply to L<DBM::Deep|DBM::Deep> as all
-Perl data structures are internally consistent.
+Perl datastructures are internally consistent.
=item * Isolated
Once the database says that a commit has happened, the commit will be
guaranteed, regardless of whatever happens. I chose to not implement this
-condition in L<DBM::Deep|DBM::Deep>N<This condition requires the prescence of
-at least one more file, which violates the original design goals.>.
+condition in L<DBM::Deep|DBM::Deep> N<This condition requires the presence of
+at least one more file, which violates one of the original design goals.>.
=back
to be successful. If you ever add a new value or if a value is used in only
certain calculations, you may forget to do the correct thing. With
transactions, you start a transaction and do your thing within it. If the
-calculations succeed, you commit. If they fail, you rollback and try again. If
-you're thinking that this is very similar to how SVN or CVS works, you're
-absolutely correct - they are transactional in exactly the same way.
+calculations succeed, you commit. If they fail, you rollback and try again.
-=head1 How it happened
+If you're thinking that this is very similar to how Subversion (SVN) or CVS
+works, you're absolutely correct - they are transactional in exactly the same
+way.
-=head2 The backstory
+=head1 How it happened
The addition of transactions to L<DBM::Deep|DBM::Deep> has easily been the
-single most complex software endeavor I've ever undertaken. The first step was
-to figure out exactly how transactions were going to work. After several
-spikesN<These are throwaway coding explorations.>, the best design seemed to
-look to SVN instead of relational databases. The more I investigated, the more
-I ran up against the object-relational impedance mismatch
-N<http://en.wikipedia.org/wiki/Object-Relational_Impedance_Mismatch>, this
-time in terms of being able to translate designs. In the relational world,
-transactions are generally implemented either as row-level locks or using MVCC
-N<http://en.wikipedia.org/wiki/Multiversion_concurrency_control>. Both of
-these assume that there is a I<row>, or singular object, that can be locked
-transparently to everything else. This doesn't translate to a fractally
-repeating structure like a hash or an array.
-
-However, the design used by SVN deals with directories and files which
-corresponds very closely to hashes and hashkeys. In SVN, the modifications are
-stored in the file's metadata. Translating this to hashes and hashkeys, this
-means that transactional information should be stored in the key's metadata.
-Or, in L<DBM::Deep|DBM::Deep> terms, within the Bucket for that key. As a nice
-side-effect, the entire datafile is unaware of anything to do with
-transactions, except for the key's data structure within the bucket.
-
-=head2 Transactions in the keys
-
-The first pass was to expand the Bucketlist sector to go from a simple key /
-datapointer mapping to a more complex key / transaction / datapointer mapping.
-Initially, I interposed a Transaction record that the bucketlist pointed to.
-That then contained the transaction / datapointer mapping. This had the
-advantage of changing nothing except for adding one new sector type and the
-handling for it. This was very quickly merged into the Bucketlist record to
-simplify the resulting code.
-
-This first step got me to the point where I could pass the following test:
+single most complex software endeavor I've ever undertaken. While transactions
+are conceptually simple, the devil is in the details. And there were a B<lot>
+of details.
+
+=head2 The naive approach
+
+Initially, I hoped I could just copy the entire datastructure and mark it as
+owned by the transaction. This is the most straightforward solution and is
+extremely simple to implement. Whenever a transaction starts, copy the whole
+thing over to somewhere else. If the transaction is committed, overwrite the
+original with the transaction's version. If it's rolled back, throw it away.
+
+It's a popular solution as seen by the fact that it's the mechanism used in
+both L<Data::Transactional|Data::Transactional> and
+L<Tie::Scalar::Transactional|Tie::Scalar::Transactional>. While very simple to
+implement, it scales very poorly as the datastructure grows. As one of the
+primary usecases for L<DBM::Deep|DBM::Deep> is working with huge
+datastructures, this plan was dead on arrival.
+
+=head2 The relational approach
+
+As I'm also a MySQL DBA, I looked to how the InnoDB engine implements
+transactions. Given that relational databases are designed to work with large
+amounts of data, it made sense to look here next.
+
+InnoDB implements transactions using MVCC
+N<http://en.wikipedia.org/wiki/Multiversion_concurrency_control>. When a
+transaction starts, it stores a timestamp corresponding to its start time.
+Whenever a modification to a row is committed, the modification is
+timestamped. When a transaction modifies a row, it copies the row into its
+own scratchpad and modifies it. Whenever a transaction reads a row, it first
+attempts to read the row from its scratchpad. If it's not there, then it reads
+the version of the row whose timestamp is no later than the timestamp of the
+transaction. When committing, the transaction's scratchpad is written out to
+the main data area with the timestamp of the commit and the scratchpad is
+thrown away. When rolling back, the scratchpad is thrown out.
+
+At first, this mechanism looked promising and I whipped up a couple spikes
+(or code explorations) to try it out. The problem I ran into, again, was the
+existence of large datastructures. When making large changes to a relational
+database within a transaction, the engine can store the rows within the actual
+table and mark them as being part of a transaction's scratchpad. Perl's
+fractal datastructures, however, don't lend themselves to this kind of
+treatment. The scratchpad would, in some pathological cases, be a
+near-complete copy of the original datastructure. N<Funnily enough, this is
+yet another example of the object relational impedance mismatch
+(http://en.wikipedia.org/wiki/Object-Relational_impedance_mismatch).>
+
+=head2 The subversive approach
+
+Despairing, I went to YAPC::NA::2006 hoping to discuss the problem with the
+best minds in the Perl community. I was lucky enough to run into both Audrey
+Tang (author of Pugs) and clkao (author of SVK). In between talks, I managed
+to discuss the problems I'd run into with both of them. They looked at me
+oddly and asked why I wasn't looking at Subversion (SVN) as a model for
+transactions. My first reaction was "It's a source control application. What
+does it know about transa- . . . Ohhhh!" And they smiled.
+
+Like Perl datastructures, a filesystem is fractal. Directories contain both
+files and directories. Directories act as hashes and a files act as scalars
+whose names are their hashkeys. When a modification is made to a SVN checkout,
+SVN tracks the changes at the filename (or directory name) level. When a
+commit is made, only those filenames which have changes are copied over to the
+HEAD. Everything else remains untouched.
+
+Translating this to hashes and hashkeys, this implies that transactional
+information should be stored at the level of the hashkey. Or, in
+L<DBM::Deep|DBM::Deep> terms, within the bucket for that key. As a nice
+side-effect, other than the key's datastructure within the bucket, the entire
+datafile is unaware of anything to do with transactions.
+
+=head2 The spike
+
+Spikes are kind of like a reconnaissance mission in the military. They go out
+to get intel on the enemy and are explicitly not supposed to take any ground
+or, in many cases, take out of the enemy forces. In coding terms, the spike is
+code meant to explore a problemspace that you B<will> throw away and
+reimplement.
+
+As transactions were going to be between the bucket for the key and the
+datapointer to the value, my first thought was to put in another sector that
+would handle this mapping. This had the advantage of changing nothing except
+for adding one new sector type and the handling for it. Just doing this got me
+to the point where I could pass the following test:
my $db1 = DBM::Deep->new( $filename );
my $db2 = DBM::Deep->new( $filename );
$db1->begin_work();
- is( $db1->{abc}, 'foo' );
- is( $db2->{abc}, 'foo' );
-
$db1->{abc} = 'floober';
is( $db1->{abc}, 'floober' );
is( $db2->{abc}, 'foo' );
-Just that much was a major accomplishment. The first pass only looked in the
-transaction's spot in the bucket for that key. And, that passed my first tests
-because I didn't check that C<$db1-E<gt>{abc}> was still 'foo' I<before>
-modifying it in the transaction. To pass that test, the code for retrieval
-needed to look first in the transaction's spot and if that spot had never been
-assigned to, look at the spot for the HEAD.
+Just that much was a major accomplishment.
-=head2 The concept of the HEAD
+=head2 Tests, tests, and more tests
-This is a concept borrowed from SVN. In SVN, the HEAD revision is the latest
-revision checked into the repository. When you do a local modification, you're
-doing a modification to your copy of the HEAD. Then, you choose to either
-check in your code (commit()) or revert (rollback()).
+I was lucky that when I took over L<DBM::Deep|DBM::Deep> that Joe Huckaby
+(the original author) handed me a comprehensive test suite. This meant that I
+could add in transactions with a high degree of confidence that I hadn't
+messed up non-transactional uses. The test suite was also invaluable for
+working through the various situations that transactions can cause.
-In L<DBM::Deep|DBM::Deep>, I chose to make the HEAD transaction ID 0. This has several
-benefits:
+But, a test is only as good as the test-writer. For example, it was a while
+before I realized that I needed to test C<is( $db1-E<gt>{abc}, 'foo' )>
+I<before> modifying it in the transaction.
-=over 4
+To pass that test, the code for retrieval needed to look first in the
+transaction's spot and if that spot had never been assigned to, look at the
+spot for the HEAD. While this is how SVN works, it wasn't an immediately
+obvious test to write.
-=item * Easy identifiaction of a transaction
+=head2 The HEAD
-C<if ( $trans_id ) {}> will run the code if and only if we are in a running
-transaction.
+In SVN, the HEAD revision is the latest revision checked into the repository.
+When you do a local modification, you're doing a modification to your copy of
+the HEAD. Then, you choose to either check in (C<commit()>) or revert
+(C<rollback()>) your changes.
-=item * The HEAD is the first bucket
+In order to make the code work for the base case (no transaction running), the
+first entry in the transaction sector became the HEAD. Thus, it was assigned
+transaction ID 0. This also had the really neat side-benefit that C<if (
+$trans_id ) {}> will run the code if and only if L<DBM::Deep|DBM::Deep> is
+in a running transaction.
-In a given bucket, the HEAD is the first datapointer because we mutliply the
-size of the transactional bookkeeping by the transaction ID to find the offset
-to seek into the file.
+=head2 Ending the spike
-=back
+At this point, I had
=head2 Protection from changes
projects / dbsrgits/DBM-Deep.git / commitdiff