[bind10-dev] Datasources, meta-datasource and configuration

[Michal 'vorner' Vaner]

Hello

We decided we need some kind of more global configuration for datasources and
that we need to aggregate more datasources so the components are able to work
with them.

First, I see several requirements around this:
 • We need a possibility to describe different data sources in the
   configuration (different types or just different sqlite3 files or whatever).
 • Multiple components need access to them.
 • Describe a way to cache some zones from some data sources to in-memory in
   auth, but not do it in xfrin/out/ddns.
 • Describe behaviour of finding the right zone to answer.
 • Aggregate the data sources together and be able to chose the right one.
 • Share the code across all modules that need it.

First, let's look at the configuration. I think we want a new top-level
pseudo-module called „datasources“. It would look something like this (well,
this is not JSON so it is more readable in the email).
datasources:
  match-type: best-match/first-match/single
              # This would describe the method of matching a zone. Best-match
              # would walk all datasources and find one that matches the most
              # labels. The first-match would pick the first zone matching a
              # little bit at least. Single would mean there's only one data
              # source to be considered (the first one).
  sources: [
     # List of the datasources. It is a list, so we can do the
     # first-match by it.
     {
       type: "sqlite3"
       params: {
               # Params would be type-specifig and optional
               file: "/var/db.sqlite3"
       }
       cache-to-mem: {
         "IN": {
           "example.org.": null # These would allow adding parameters to cached
                                # zones
           "example2.org.": null
         }
       }
     },
     {
       type: "zonefiles" # This one is virtual to load data to in-memory only.
                         # It looks empty to other uses.
       cache-to-mem: {
         "IN": {
           "example.com": "/path/to/zonefile"
         }
       }
     }
     {
       type: "memory"
       # Only existence (and position in the list) is noted here, it is
       # filled based on the other datasources' cache-to-mem entries.
     }
 ]

The other way to fill the in-memory would be to give a name to each of the data
sources and list the zones to load in the in-memory data source configuration,
with names of the data sources to load from.

We could have a meta data source. That would do only one thing ‒ hold bunch of
sub-datasources. Whenever someone asked it, it would find the corresponding one
(depending on the type of match), take the zone finder and give that one
directly. It wouldn't need to have its own data source.

Then there would be a factory that would take the config, create the data
sources and either return single one as a result (if there was only one or the
match-type is single) or a meta datasource build of the list. The memory data
source would be skipped in the xfrin/xfrout/etc (and not loaded, of course).
Maybe we want two type matches, one for auth and one for the rest (so we can
have single in the auth and best-match for xfrin/out, as the single one would be
memory-only cache of everything, but ignored in xfrin/out).

The single match type could be used, for example, if we have sqlite3 for the
xfrin/out, but we want to serve from memory only. So we do something like:
"datasources": {
  "match-type": "single",
  "sources": [
    {
      "type": "memory"
    },
    {
      "type": "sqlite3",
      "cache-to-mem": {
        "IN": {
          "example.org": null
        }
      }
    }
  ]
}

Do you think it makes sense?

With regards

-- 
No, I will not change your lightbulb.

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20120418/f4cd1c14/attachment.bin>