[bind10-dev] initial ideas on the "difference" design

[JINMEI Tatuya / 神明達哉]

At Wed, 12 Oct 2011 12:07:20 +0200,
Michal 'vorner' Vaner <michal.vaner at nic.cz> wrote:

> > Then the stored database rows corresponding to the above example would
> > look like this (assume "zone_id" for example.com is 10):
> > ID ZID  ver op  name            rrtype
> > 1,  10,   1, 0, "example.com.", "SOA",...
> > 2,  10,   1, 0, "a.example.com.", "A", ...
[...]
> What is the reason for having the ID? We don't need to reference exact row of
> database ever, so there's no need to have an identifier and I guess the order
> inside the diff does not matter provided deletions are first, right? Because the
> diff is atomic anyway.

Two reasons: first, this representation is intended to be used as a
valid sequence of IXFR, so at least the first RR of each
"delete"/"add" part must be SOA.  Using IDs and sorting results by ID
are one convenient way.  Second, this representation assumes "version"
is the SOA serial, so we cannot assume that versions monotonically
increase and that we can compare them just like bare integers.  On the
other hand, if these points can be addressed in a different way, it
doesn't have to be IDs.  (e.g. for the first point, we may want to
achieve this by introducing 4 different operation types: 0: delete
SOA, 1: delete other RRs, 2: add SOA, 3: add other RRs, and sort them
in this order).

> Also, from the interface point of view, I'd say the journaling would be allowed
> to write the diffs even if the updater's parameter is False (eg. git-based data
> source, where each change is a commit, it generates any differences whenever you
> ask).

I'm afraid I don't understand this...do you mean it should be possible
for a data source implementation to record diffs anyway if the higher
level begins with "ZoneUpdater(zone, journaling=False)"?  If so, I
don't think it's prohibited by this design.  This design only requires
diffs must be logged if journaling is True, but doesn't say they must
not be logged if it's False.

> > select * from diffs where
> >   zone_id = Z and
> >   id >= (select id from diffs where version = B and operation = 0
> >          order by id asc limit 1) 
> >   and
> >   id <= (select id from diffs where version = E and operation = 1
> >          order by id desc limit 1);
> 
> Maybe:
>   select * from diffs where
>   zone_id = Z and
>   version >= B and version <= E order by (version, operation);
> 
> (With some tweaks, of course). This one looks less complicated and might be
> faster. This expects the version, zone id and operation are indices (not
> necessarily primary ones).

See the first point.  We cannot simply do 'version >= B' or 'version
<= E' unless we convert SOA serials to monotonically increasing
"versions".

But I'd note that the specific schema and SQL statement to retrieve
the sequence is just one example approach anyway.  As long as it meets
the requirements of the higher level APIs, the internal representation
is flexible.

> I'm not sure I'd like a de-coupling of diff-commit from the normal commit. What
> happens if the first succeeds and the second does not? Should we delete the
> diff? Or try to re-apply it? Or call for help? I think both storing the diff and
> the data themself should be part of the same transaction and become atomic.

My intent was "it depends on the implementation".  The only
requirement is that when the normal commit succeeds the diff-commit
must always succeed (or have succeeded by then).  This is derived from
the BIND 9's journal design: In BIND 9 the journal is a persistent
storage, while the (run time) zone data storage is in-memory and
volatile.  So when BIND 9 named updates a zone it must first ensure
diffs are committed to the journal.  If subsequent update operations
on the actual update fail, named will keep running with the old zone
data, and will apply the diff to the actual zone next time it starts
up.

Of course, whether it makes sense to apply this operation mode to BIND
9 is a different question, and I'm actually not so sure about that.
One thing I'd note is that we'd basically use the same backend
database for the diffs and zone data, in which case we use the same
single transaction, and the entire updates for diffs and zone should
be atomic.

In any case it's a good question what it means if diff commit succeeds
and the normal commit fail.  If we allow this to happen, we should
probably differentiate the results (either using different exceptions
or different return codes or some other ways) so the caller can react
accordingly if it wants.

---
JINMEI, Tatuya
Internet Systems Consortium, Inc.