[bind10-dev] Data source API

[Danny Mayer]

Michael Graff wrote:
> Here's some quick thoughts on how I'd like to approach the data source
> API in the short and near term.
> 
> Initially, it will be very simple and may only allow for very simple,
> low-level functions:
>     Look up data based on (name, type)
>     Update data
> Step two will be:
>     Return associated data (like CNAME targets, DNSSEC records, etc)
> 
> I envision a 3-layer approach eventually:  the application should have
> to do very little "grunt work".  The database layer itself will do most
> of the hard stuff, since the closer we put things like searching to the
> data itself, the faster and more efficient I think it will be.  A middle
> layer will be in place to implement searching and other things that a
> specific layer may not implement.
> 
> Initially this will be a very shallow design, with the application doing
> more than it should.  This middle layer should grow out of refactoring
> work as we add more than one data source, and find that we are doing
> common tasks which can move up from the low level, and as we write more
> applications/tools which use the data sources, and functionality moves
> from the apps into this middle layer.
> 
> Short-term design would:
> 
>   (1)  Work on the RRSet level, not on the record level.
>   (2)  Be fast for single-name/type lookups.
>   (3)  May be a read-from-disk, in memory database, and updates will
>        go to disk periodically.
>   (4)  Perhaps going straight to sqlite is just better.  It's really
>        fast, and can be used entirely in-memory.
> 
> The important thing here is I don't want to do what BIND 9 did, if we
> can avoid it -- we are not database experts, so using pre-written data
> structure libraries is likely better for us initially than rolling our
> own.  I hope it is better for the long term too.

If you need database experience I'd be happy to lend a hand at advising
you on that. Most of my current experience has been with Oracle and MS
SQL Server, but most of it applies to whatever database you may choose
to use. I've spent a lot of time over the past years tuning queries (you
won't believe the kind of stupid architectural issues that I've had to
fix). The most important thing with a database (like anything else such
as BIND10) is the get the underlying architecture of the tables right.
You can add indexes as you go but you need to remember the changes being
made to tables affect indexes and lead to fragmentation and therefore
performance issues. And yes you can fragment your indexes as well. So
whatever you use for a backend database needs to not only be able to
deal with fragmentation issues but also defragmentation while the
database is in use. It also needs a good backup/restore as well as
transaction logging in case of database failures.

You also need to be able handle database errors, including connection
failures.

Danny