[bind10-dev] proposed design of scalable in-memory zone loading/updating

[Jerry Scharf]

strong +1 for copy on write. doubling the memory footprint seems like a 
bad thing.

jerry

On 06/25/2012 06:02 AM, Michal 'vorner' Vaner wrote:
> Hello
>
> On Fri, Jun 22, 2012 at 11:13:19AM -0700, JINMEI Tatuya / ???? wrote:
>>> However:
>>> . The "MemoryEventManager" seems to suggest there's a main-loopish thing hidden
>>>    inside and that it'll dispatch events somehow.
>> First, it's "MemoryEventHandler", not manager.  And the event loop is
>> supposed to be outside of the handler.  So, I'm not really sure about
>> your concern, but if you mean the naming is confusing (even after
>> clarifying it's not named "manager"), I'm open to suggestions.  I'm
>> not very good at naming things.
> Yes, I meant the naming confusion. Maybe something like SegmentManager?
>
>>> . I'm little bit worried about the auth pushing the updates to the in-memory
>>>    clients. When we switch to the new data sources, the in-memory clients would
>>>    be effectively hidden from auth and there'd be multiple ones (as caches for
>>>    multiple data sources). I think we'll need an other way around - each
>>>    in-memory contacts the manager and asks for data for the zones which should be
>>>    inside (and, possibly, updates).
>> Yeah I realized that and considered several other options for the
>> cleanest (or least ugly) design.  The currently resulting one is a
>> kind of compromise.
> I was thinking the auth would still call the manager. But each in-memory would
> register itself within the manager and when the manager has a new segment or
> something, a callback of the in-memory is called instead of returning it to auth
> and auth pushing it to the in-memory.
>
>> Hmm, in the case of IXFR (or DDNS, or in general "update", not
>> "replacement"), I thought the local handler makes the update directly
>> to the "in-use" data (on the other hand the shmem-based memory manager
>> would keep two copies and always make modifications to the unused
>> one).  So this update process cannot be done incrementally; otherwise
>> queries in the middle of the update could be handled using partially
>> updated version of zone.
> Ah, I see.
>
> Also, the manager, would it be possible to have some kind of copy-on-write
> memory pages, so we don't have to keep two full copies?
>
>> Aside from these difficulties, however, always doing it asynchronously
>> may make sense; then the caller side of code can be unified.
> It is maybe not worth it, when you pointed out the problem with partial update.
>
>>> . Would releasing really be time consuming? Wouldn't just dropping the memory
>>>    segment be enough? That would be fast.
>> I'm afraid it can be time consuming at a non-negligible level if it's
>> built locally as we'll need to go through the entire tree and the node
>> data, releasing corresponding memory chunk one by one.  Applying an
>> analogy with experience with BIND 9, that could be 10sec-ish task.
>>
>> ...or, perhaps you are assuming we use process-local, exclusive memory
>> segments preallocated by mmap or something?
> Yes. In that case, we could just munmap it and not worry about what was inside.
>
>>> But I'm most worried about the continuation objects (which would be,
>>> effectively, coroutines). That could get really tricky (because we'll need to
>>> remember state somehow, etc). In short, I consider working with continuation
>>> objects a really interesting part of programming, but it might be better to
>>> avoid it. So I'm wondering two things:
>>> . Would it make more sense to launch a thread? Because the thread would do no
>>>    communication with the rest of the app, so it would be safe, we wouldn't need
>>>    much locking (only to check if it ended already). Then we could do the loading
>>>    in a more straight-forward sequential logic.
>> Hmm, possibly.  And, considering that the memory handling part of the
>> local handler is actually emulating an external "manager", it may be a
>> better instantiation.
>>
>> One obvious concern is the "update" case (see the IXFR-case discussion
>> above).  If the updating thread modifies the zone data used by the
>> main auth thread (responding to queries) the contention will cause a
>> disaster.  We should (probably) also think about the case where
>> multiple replacement requests happen at the same time (may or may not
>> be an issue).
> These don't have the continuation object, so they would run synchronously,
> without a thread.
>
> With regards
>
>
>
> _______________________________________________
> bind10-dev mailing list
> bind10-dev at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind10-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20120625/59145ee7/attachment.html>