[bind10-dev] configuration diffs

[Jelte Jansen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


(tl;dr: let's ditch the "here's some changes, you figure it out"
approach, and replace it with operations and callbacks)


A number of problems (or inconveniences) we are currently having
regarding configuration is the fact that we don't really have an
internal concept of 'differences'; for basic types this is not a
problem (the change is the new value), for normal maps we do, kind of
(any entry in a map on an update means it has changed; a null entry
means it has been removed, and a 'missing' entry means it has not
changed, but even this hasn't really been formalized). And what is
actually sent around is a sort of 'overlay', which when applied merged
into the existing data structure results in the new one. This has some
advantages (relatively easy bookkeeping, and low fixed overhead), but
also some disadvantages (extensibility is low without adding lots of
complexity, lots of local overhead to check data)

And for lists we don't have anything at at all right now, so every
list change causes lists to be copied, and reinterpreted completely.
The same goes for named_sets, which are essentially a different type
of list.

So we need something for configuration update handlers to know what
has changed (added/removed/updated), and we need a representation for
that. For the sake of argument, and to show the problem, I'll start
out with the lowlevel representation.

I have been thinking about this and experimenting with an approach
today, to see if we can incrementally add this without doing some of
the more fundamental changes we discussed at the F2F. Unfortunately, I
fear the former may be too much of a dead end without the latter.

To show why, let me first focus on lists and their representation when
sending them around.

As said, currently, when a list (or one of its elements) is changed,
the entire list is copied, and any existing element is removed, then
added from the 'new' list; you add a zone, and all zones are read anew.

So one idea is to send around not a direct list, but a map, or a list
of lists, which contain 'added' and 'removed' values;

For example, if you have a data element of type list, and you get an
update, the updated value is either:
A) a list itself; signaling that the list is to be replaced completely
B) a map containing:
   1) "removed": <list of removed items, complete matches>
   2) "added": <list of added items, complete matches>

The handler could then check whether it needs to replace all (case A),
or leave existing stuff alone and only remove certain values (B1)
and/or add them (B2).

The problem, then, would be in nesting; if we have a list of lists,
and add an item to the inner one, how do we represent the outer list?
The list itself hasn't really changed, but some of its elements have.
So either we need to then remove and readd the changed outer element
(which would also cause unnecessary processing), or we could have to
add another option 'changed'; the elements in changed should already
exist, but have child elements that have changed. And the full list
(A) option would probably have to mean 'no additions or removals, just
updates'. Maybe we shouldn't even have it in the first place.

The next step would be a way to report back which operations succeeded
and which operations failed (so you could say 'of those ten zones you
added, 8 succeeded, but i had a problem with these 2, so i did not
load them'). The config manager could store that, and set 'current
config' to the succeeded one, then report back which 2 failed, so
bindctl (or any other tool) can then update locally that the two
failures have not been committed.

Nice in theory, but given that we are talking about data that should
be synchronized, this all gets more and more fragile. Even more so
since all these moving parts work with 'raw' data (Element objects on
c++ side, and basic types in python). It was already kind of over the
limit of readability there, and this would probably push it deep into
mount doom itself.

And this is just for basic lists...

So I think that we cannot do this reliably without some of the other
fundamental changes, starting with the 'overlay' approach that is
currently used, and rather send around a set of operations that are to
be done (to which the components that handle them can then tie actions
if they want).

So, for instance, instead of a general ConfigHandler method, one would
set up actions to be called when values are changed, added, or
removed; should one not want to do that, I would still like to be able
to read data directly (from 'current' config).

And while we are meddling with that, we might as well fix the defaults
issues.

I have not thought up an API for this yet (there is some serious
designing to be done there), probably a good idea to argue about the
general idea first :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+xGu0ACgkQ4nZCKsdOncXaDACfcNC8eeLv462Qxt5h31vKXGYx
kZAAoLunaIld97hGUuGHQCWZgCHbTRfb
=PATD
-----END PGP SIGNATURE-----