[bind10-dev] new in-memory zone data design

[Michal 'vorner' Vaner]

Hello

On Wed, Jun 20, 2012 at 03:22:21PM -0700, JINMEI Tatuya / 神明達哉 wrote:
> I agree it's not super simple, but I'm not sure specifically which
> part you think was hell.  Possibilities are:
> 
> 1. how to encode names in tree nodes
> 2. how to encode RDATA
> 3. how to maintain and use of shortcut

Not any specific point, it just gave an impression of large and complex system.

> >  * What about memory management? What we do with a space where we deleted
> >    something? Just keep track of how much „wasted“ space we have and then
> >    recreate the structure when too suboptimal?
> 
> I'm not sure if I understand the question.  The initial version will
> just use the C++ new/delete.  So a deleted space won't be a waste
> (modulo fragmentation problems with new/delete).  If and when we
> successfully switch to a shared-memory version, we use a dedicated
> allocator for the shared memory region (we can begin with the one that
> comes with boost managed_shared_memory initially, and if it's not
> efficient we can then think about an in-house version).  These
> allocators will also recycle de-allocated memory region.  The
> different from the normal new/delete is that we'll eventually hit the
> originally allocated space for the shared memory segment.  At that
> point we'll need to do a non trivial task - reallocating a completely
> new space then copy the image, etc.  I've not fully thought about
> these advanced cases, but is such a case your question?

It more or less does answer. But in that case, we may want to use the allocator
(or some wrapper of it) right from the beginning, so we don't have to go hunting
through the code and find all places where we allocate.

I don't think there'll be much problem on reallocation. The worst that can
happen is we need to allocate and do a memcpy (since the image should be
position-independent), which will take some time, but not extra too much.
Provided it is in the external tool, we won't stop answering queries.

But linux, and I hope many other systems, have calls like mremap, which is able
to extend or shrink a memory mapped region without copying the data. If there's
not enough virtual memory behind, it places it at a different virtual memory
position, but the physical data stay on the same place. So that one is pretty
fast.

> >  * Again, when we delete something and something else pointed to it, what
> >    happens?
> 
> With reference counting, we cannot delete anything as long as
> something points to it.

So, are we going to do our own manual reference counting? Or can we combine the
offset pointers with shared_ptr somehow?

> >  * Is it possible to deduce the name from the node only, or do we need the
> >    path?
> 
> It's not possible without the path in the initially proposed version.
> Depending on a tradeoff between memory efficiency and processing
> efficiency, there can be other choices.

So, if there's a shortcut-link to another node, how do we know the name?

Thank you

With regards

-- 
I'm reading your disk
		-- General Failure

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20120621/41aa441b/attachment.bin>