[bind10-dev] some other details about differences

[JINMEI Tatuya / 神明達哉]

I have some more details to discuss regarding the differences, which I
omitted in the previous thread as that one was already pretty large
(and it wouldn't have made much sense to discuss too many details
before agreeing on the basic concept).

These are independent topics, so feel free to make
comments/feedback/suggestions on any one of them, ignoring others.

1. Housekeeping

For a longer term the diffs table will get larger and larger, and I
guess we should eventually worry about its size.  In BIND 9 there's a
configuration max-journal-size, and if it's set to a finite value
named will keep the size of journal files so that the file size won't
exceed the specified maximum.  In the approach of using a database
table, we cannot use a "file size", so we probably use the max number
of versions that can be stored.  We'll probably also need a separate
housekeeping process (which maybe co-located on some existing b10-
program) that manages the size of diffs for each zone.

One possible nice thing of the database approach is that it would be
relatively easy for the administrator to manually drop some older
versions of diffs, e.g., by 
selete from diffs where
  id <= (select id from diffs where version = X and
         operation = 1 (add)
	 order by id desc limit 1);

2. Special support in dynamic update case

The proposed internal representation assumes IXFR-style sequences (old
SOA, deleted RRs, new SOA, added RRs).  Obviously this is convenient
to store diffs from ixfr-in and retrieve diffs for ixfr-out.  But in
the case of dynamic update we need some more work to ensure the stored
diffs are an expected sequence.  BIND 9 provides an additional
interface to write a set of diffs at once, which internally normalizes
and sorts the diffs into the expected sequence, and dump the diffs to
the journal file in a transaction.

We could take a similar approach:
class ZoneUpdater():
...
    def write_diffs(self, diffs):
        sorted_diffs = sort_diff(diffs)
        # Note: diff transactions may be NO-OP if the implementation chooses
        # the transaction started by the updater itself.
        self.__accessor.begin_diff_transaction()
        for diff in sorted_diffs:
            self.__accessor.add_diff((operation, diff))
        self.__accessor.commit_diff()

Alternatively we can publish the sorting method, and have the update
process sort the diff and make updates in the sorted order (with
journaling=True).

3. Propagation to in-memory data source

Another issue is how to propagate the updates to an in-memory data
source when the in-memory internally uses database-based data source
(the target of this propagation may be either the auth process or
something like a "shared-memory manager" if we use this technique for
in-memory, but this level of details don't matter much in this
discussion).

One trivial way is to have the target process retrieve the diffs from
the database and apply it to the in-memory image.  This may work, but
I'm a bit worried about its performance implication if it's running
with a high volume of updates (e.g. over several thousands of
updates/sec).  We might want to use a direct channel between
xfrin/DDNS process and the process managing the in-memory image, e.g.,
via the command control channel.

Right now I'm not sure if this concern is real, and if so what's the
best way to deal with that.

---
JINMEI, Tatuya