[bind10-dev] proposed design of scalable in-memory zone loading/updating

[Michal 'vorner' Vaner]

Hello

On Fri, Jun 22, 2012 at 11:13:19AM -0700, JINMEI Tatuya / 神明達哉 wrote:
> > However:
> > • The „MemoryEventManager“ seems to suggest there's a main-loopish thing hidden
> >   inside and that it'll dispatch events somehow.
> 
> First, it's "MemoryEventHandler", not manager.  And the event loop is
> supposed to be outside of the handler.  So, I'm not really sure about
> your concern, but if you mean the naming is confusing (even after
> clarifying it's not named "manager"), I'm open to suggestions.  I'm
> not very good at naming things.

Yes, I meant the naming confusion. Maybe something like SegmentManager?

> > • I'm little bit worried about the auth pushing the updates to the in-memory
> >   clients. When we switch to the new data sources, the in-memory clients would
> >   be effectively hidden from auth and there'd be multiple ones (as caches for
> >   multiple data sources). I think we'll need an other way around ‒ each
> >   in-memory contacts the manager and asks for data for the zones which should be
> >   inside (and, possibly, updates).
> 
> Yeah I realized that and considered several other options for the
> cleanest (or least ugly) design.  The currently resulting one is a
> kind of compromise.

I was thinking the auth would still call the manager. But each in-memory would
register itself within the manager and when the manager has a new segment or
something, a callback of the in-memory is called instead of returning it to auth
and auth pushing it to the in-memory.

> Hmm, in the case of IXFR (or DDNS, or in general "update", not
> "replacement"), I thought the local handler makes the update directly
> to the "in-use" data (on the other hand the shmem-based memory manager
> would keep two copies and always make modifications to the unused
> one).  So this update process cannot be done incrementally; otherwise
> queries in the middle of the update could be handled using partially
> updated version of zone.

Ah, I see.

Also, the manager, would it be possible to have some kind of copy-on-write
memory pages, so we don't have to keep two full copies?

> Aside from these difficulties, however, always doing it asynchronously
> may make sense; then the caller side of code can be unified.

It is maybe not worth it, when you pointed out the problem with partial update.

> > • Would releasing really be time consuming? Wouldn't just dropping the memory
> >   segment be enough? That would be fast.
> 
> I'm afraid it can be time consuming at a non-negligible level if it's
> built locally as we'll need to go through the entire tree and the node
> data, releasing corresponding memory chunk one by one.  Applying an
> analogy with experience with BIND 9, that could be 10sec-ish task.
> 
> ...or, perhaps you are assuming we use process-local, exclusive memory
> segments preallocated by mmap or something?

Yes. In that case, we could just munmap it and not worry about what was inside.

> > But I'm most worried about the continuation objects (which would be,
> > effectively, coroutines). That could get really tricky (because we'll need to
> > remember state somehow, etc). In short, I consider working with continuation
> > objects a really interesting part of programming, but it might be better to
> > avoid it. So I'm wondering two things:
> > • Would it make more sense to launch a thread? Because the thread would do no
> >   communication with the rest of the app, so it would be safe, we wouldn't need
> >   much locking (only to check if it ended already). Then we could do the loading
> >   in a more straight-forward sequential logic.
> 
> Hmm, possibly.  And, considering that the memory handling part of the
> local handler is actually emulating an external "manager", it may be a
> better instantiation.
> 
> One obvious concern is the "update" case (see the IXFR-case discussion
> above).  If the updating thread modifies the zone data used by the
> main auth thread (responding to queries) the contention will cause a
> disaster.  We should (probably) also think about the case where
> multiple replacement requests happen at the same time (may or may not
> be an issue).

These don't have the continuation object, so they would run synchronously,
without a thread.

With regards

-- 
The problem with graduate students, in general, is that they have
to sleep every few days.

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20120625/0ffab1cb/attachment.bin>