[bind10-dev] [jinmei at isc.org: Re: proposed system design and inter-process protocol for shared memory]

[Michal 'vorner' Vaner]

Hello

Bouncing didn't work, so I'm trying to forward instead.

----- Forwarded message from JINMEI Tatuya / 神明達哉 <jinmei at isc.org> -----

Date: Wed, 06 Mar 2013 22:59:42 -0800
Message-ID: <m27gljoftt.wl%jinmei at isc.org>
From: JINMEI Tatuya / 神明達哉 <jinmei at isc.org>
To: Michal 'vorner' Vaner <michal.vaner at nic.cz>
Subject: Re: [bind10-dev] proposed system design and inter-process protocol for shared memory
In-Reply-To: <20130306090856.GA21243 at ruth>

At Wed, 6 Mar 2013 10:08:56 +0100,
Michal 'vorner' Vaner <michal.vaner at nic.cz> wrote:

> There's something I'd strongly disagree with. The design is a fine example of
> how we try to babysit the messaging system and work around some missing parts,
> instead of fixing it. This is not the way forward, since, as you admit, the
> protocol is complicated and it is needlessly so.
> 
> So, first, our messaging system uses TCP. That is, deliver or die principle. You

A minor correction first: it doesn't use TCP; it's IPC between UNIX
domain sockets.  I'm not really sure if any standard (POSIX?)
guarantees it's reliable, but at least it seems possible to assume
that as a matter of de facto.  So the rest of the discussion holds
whether it's TCP or UNIX domain IPC.

> can never lose a message in the middle of TCP stream. We also say we consider
> death of the connection an unrecoverable error, since we could lose messages in
> between. If a module loses connection to msgq, it crashes and it is for this
> purpose.

I'm afraid this might be too harsh.  We've been seeing some strange
network errors on the UNIX domain sockets, so such cases do not seem
to be super exceptional (and it's not clear whether they are
recoverable or fatal).  In the C++ case additional layers of ASIO
makes it more complicated.  But, at least for the initial
implementation I can live with that assumption and seeing how it goes.
We should include this requirement as part of the API contract (e.g.,
require applications not to catch and handle some specific exception
but to terminate).

> If msgq itself dies, then whole bind10 goes down. The only way a
> message might not be delivered to a recipient is when the recipient dies (or
> enters an infinite loop). The first needs to be solved (see below), the second
> won't be solved by retransmits anyway. So, please don't require retransmits,
> retransmits over TCP are wrong.

I knew we can simplify the protocol if we could clarify the
reliability of the message system (depending on how we clarify it).
The initially proposed one was based on the assumption that we won't
have time for it with accompanying extensions.  So the question we
should discuss is whether we include these tasks in this shared-memory
feature.  It depends on the expected amount of work, but personally I
think it's better to include them.  These clarifications and msgq
extensions such as the ability of getting a list of live components
will be necessary for other purposes, and will make help the overall
system more robust.  But we should discuss this project-wide.

With this approach, I suspect we also need to ensure the high level
"send message" operation is non-blocking.  It's now critical since at
the lower level it can block, and the application may not wait, and we
now cannot simply let it fail and reconnect (because it now means
suicide).  We should implement it with an intermediate buffer +
internal asynchronous write, or by even introducing more generic
asynchronous message exchanges and using it.

And, I wonder whether we can totally eliminate timers.  For example,
if a user/consumer process has a bug and does not correctly respond to
an update mapping notification, the entire system will be effectively
in a deadlock (in terms of shared memory management).  Even if we can
detect it with a timer there may not be a way of graceful recovery,
but I guess it's still better than a deadlock like situation.  That
can be skipped in the initial implementation, though.

> This is a counter-proposal:

From a quick read it looks workable.  There are some things I didn't
fully understand:

- what's "the consumer channel"?
- how can msgq/manager/consumers distinguish them?  In particular, how
  can the manager distinguish individual consumers?
- how does msgq know it when a component (e.g. auth) crashes?  And
  how does it tell another component (e.g., memmgr) that?

---
JINMEI, Tatuya
Internet Systems Consortium, Inc.


----- End forwarded message -----

-- 
In the name of kernel, compiler and holy penguin

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20130307/b946dbd2/attachment.bin>