[bind10-dev] proposed design of scalable in-memory zone loading/updating

[Shane Kerr]

Jinmei,

On Thursday, 2012-06-21 23:57:30 -0700,
JINMEI Tatuya / 神明達哉 <jinmei at isc.org> wrote:
> It's described in the wiki page:
> http://bind10.isc.org/wiki/ScalableZoneLoadDesign

Some questions...

Loading after startup is to be done in "chunks" of some reasonable
size, presumably. I guess this is something we can limit based on
wall-clock time to insure reasonably low delays, right? So, say check
the time after each 100 (or 10 or 1000) operations and pause loading
after 0.001 (or 0.01 or 0.0001) seconds have passed?

Not really related to this particular proposal, but we may want to be
careful with the IXFR event chain to insure that we've loaded our
in-memory database before we send NOTIFY packets. Even so, it is
possible that a secondary could get a newer version of the zone than we
are serving. (I am assuming that b10-xfrout reads directly from the
SQL database.) This is a truly edge case, so probably nothing worth
engineering around, but perhaps worth noting.

Like you, I also considered the idea of having a maximum size of
difference for IXFR before considering it a case for a full load.  In
theory, it may be better to have an incremental-difference reader, but
in practice I expect that this is so rarely true we don't need to
engineer a solution for it at this point. (It is good to leave future
BIND 10 developers something fun to do.) ;)

Like Michal, I also considered the idea of using threads rather than an
event-driven loading design. I think the actual work and state involved
with loading is straightforward enough that it should not be too
complicated. I don't think we want co-routines though, but rather the
non-co-routine equivalent simply maintaining load state. It lets us
avoid messy concurrency issues, locking, and so on.

A final consideration is whether we want to have multiple Memory Event
Handler instances per process. The use case here is when we are loading
a large zone and don't want to block updates to other zones. It may
complicate the design slightly, but if we don't include it now and add
it later it seems like it may be quite a bit of work to refactor around
this idea.

Cheers,

--
Shane