[bind10-dev] Configuration, bindctl, etc.

[Jelte Jansen]

OK I promised an idea dump/initiative for a roadmap, so here it is:


IMO there are currently three issues regarding configuration, and the
way we interact with it (i.e. bindctl in its current forms). There are
more things 'missing' as in 'not implemented' in order to make it Really
Awesome, but these should probably be addressed first, to make extending
easier.

Fundamentally, I see these points of improvement:
- Everything is handled from the raw data, directly as read from JSON
  (both for (partial) data itself and for specifications).
- There is no representation for 'changes' or 'diffs'
- there is no abstraction between data as the modules would want it and
  data as the admin would want it (resulting in the dreaded 'remote()'
  stuff)

Secondary, and these derive mostly from the above points:
- Any particular piece of 'data' misses its context (where it fits in
  the total configuration, and what its specification is, what its
  valid values are)
- Juggling two sets (specifications and data) around makes bindctl very
  fragile (especially in config_data)

So my general proposal is to *at first* keep the top-level as-is, but
make a number of internal changes:

- wrap actual data in a new class which provides access to:
	- the 'raw' data
	- the specification part of this data
	- the 'parent' of this data (if any) (or at least the relative
          location)
	- the 'children' of this data (if any)
	- the 'name' of the element (if applicable) that contains this
          data

instances of this class are produced by validating any set of raw data
against a specification (part), and are passed around where currently
Elements (or raw python types) are passed.

This should provide us a couple of relatively small wins:
	- finally a decent way to get defaults
	- errors can contain more context when validating
	- no need to 'address' specific data parts all the way from the
          top

This is something that would be directly usable on the module-side,
without breaking existing code.

The next step could be, while keeping current functionality, replace the
config_data guts of bindctl (and probably the original named arguments
part). TBH I currently see no much better way than to rip out all
config_data.py code, and especially the MultiConfigData class. Nearly
everything in there is about addressing and juggling the raw data of
both specs and actual data around. So I'm afraid that apart from the
actual connection part, this is mostly a rewrite.

This should make bindctl much more stable. However, if bigger bindctl-ng
plans are in the works this step could probably be skipped. (else we'd
be rewriting it twice). However, I suspect it will make the next step
much easier as well;

Then we should probably define a way to codify 'differences' (there was
a proposal somewhere for a format), and wrap those in classes as well;
and then add generate, check and apply methods to the class above, and
update config handlers to work with them.

This should for starters solve the issue of passing long lists around,
but should also make config update handlers easier.

When we have these, I think a good next step would be to make the
'global' config update a three-phase commit; instead of direct updates,
first let all modules check (yes, still 'online').

We could also use these to keep and show config history.

So far, there have been small improvements, while mostly keeping current
functionality as-is. But I do think these changes provide the foundation
needed to implement cooler functionality;

I'm just gonna list some of the ideas here, and perhaps how they could
be approached.

- 'offline' config.
    Something we've wanted for a long time. One way would be to extend
    the specification language to specify 'rules' that can be verified
    at all times (so include semantics in the specification; run-time
    problems will always be a problem for this anyway). But it may be
    tricky to make this flexible enough.
    Another approach would be to make the current 'plugin' system in
    cfmgr more of a first-class citizen, and allow these for normal
    modules as well (so when you write a module, you can additionally
    provide a small python file to do config checks).
    And a third would be very wild I guess; we change components not to
    be executables but dynamic libraries, with a few well-known
    methods, such as 'run' and 'check-config'.

- custom types
    I think that we should have more than just raw types; IP addresses
    come to mind instead of strings, but also zone names, etc. The
    immediate advantage is that (like for offline config), these are
    pretty well-defined and easy to verify.
    A second level of this would be extensible types; compound types
    that could be reused.
    Another advantage is that user interfaces could make use of this
    info to make them more userfriendly.
    An addendum to this is that with it we should be able to make
    recursive types, and not have to deal with the 'any' thing we have
    now for ACLs etc.
    Currently this is hard to add because of the way raw data is parsed
    and handled, but given the earlier changes I think this can finally
    be added.

- also related to the previous two perhaps, but we need to extend the
  specification 'language'.
    For starters, it might need a bit of context sensitivity (option B
    can be either X or Y depending on the value of option A, e.g.
    datasources config).
    Related to the first one, perhaps we can add some basic rules (also
    something suggested a long time ago, for instance for integers
    specify a valid range, for strings specify a regex that should
    match, etc.). This might also be a good time to reconsider whether
    JSON is a good format for the .spec files themselves.

- abstract data 'as needed by us (the system)' and 'data as logical to
  the administrator'.
    Sorry, nothing concrete here atm, but I still think we should
    internally keep data grouped as close to the modules as possible,
    while for administrators it may be much more convenient to group it
    on topic. Not sure how though; ideally we might even be able to
    have custom abstractions per tool, but that is just a wild dream at
    the moment.


This all still leaves out some other improvements, such as a separate
'message' channel to provide (late) feedback to bindctl, and direct
connection to msgq, etc. These are nice but IMO pretty much separate
from all this.