[bind10-dev] MSGQ-NG requirements

[JINMEI Tatuya / 神明達哉]

At Thu, 5 Apr 2012 20:18:43 +0200,
Michal 'vorner' Vaner <michal.vaner at nic.cz> wrote:

> > - I was not sure if a central server is an assumption (or is it an
> >   implementation issue?)
> 
> Well, I kind of assume it would be there, for two reasons:
>  • We have it now. I'd like to simply improve the current situation somewhat
>    incrementally, instead of implementing something completely new (as I hope it
>    would be less work with reusing what we have, or it would at least cause less
>    disruption on the way).
>  • Each module might want to communicate with many others (and, in case we
>    support notifications, it might not even know with which modules). Doing it
>    without a central server might be possible if we really insist, but I believe
>    it would be much more complicated.
> 
> Is there a reason not to have a central server?

Not much, these were generally random questions I happened to have for
clarification.  But if we want to use the msgq-ng for transferring
"socket sessions" (FD + data) as well as for the current configuration
+ command parameters, it would be a bit more specific and serious
question due to both the complexity and performance considerations.

> > - Regarding high-load support, I wonder whether we have any expected
> >   scale where this generic msgq is applicable, or whether it's
> >   "infinite".  I also wonder performance requirement in terms of
> >   transaction speed.
> 
> By transaction speed, you mean the round-time trip? I kind of assumed the msgq
> would be mostly idle (while the other components would be doing the real work),
> and the communication happening on the local message, so it would not be a
> problem. I'd be more worried about the speed in which a remote component is able
> to answer.
> 
> And by the scale, I believe if the server is able to handle 1 000 000 zones, the
> MSGQ should not be the bottleneck to prevent it. Otherwise, I don't think it
> needs to be infinite, I just don't have the exact numbers.

Beyond some reasonably large point, a very large scale is effectively
"infinite" in the context of my question.  So I interpret we generally
assume an infinite capacity in msgq.

Whether this (both speed and scale) matters depends on for what and
how we want to use it.  Are we assuming we'll pass a list of the 1
million zones with additional per-zone configuration via msgq?  What
if the zones are primarily configured in a database backended data
source?  What about ACLs, especially if they are defined per zone?

As for the speed, if we pass that amount of data over msgq with
involving corresponding JSON parsing and validation, I'm afraid it
could affect non-negligible impact on, e.g., the start up time.
Partly due to these concerns and partly because of the (apparent) fact
that the primary storage of a list of served zones is in a separate
database system at least for database-based data sources, my general
assumption was we'll only exchange higher level information over msgq
such as a list of data sources used, thereby making the scalability
concerns with msgq less critical.

I'm not necessarily insisting that's the definite way to go, but
apparently we are having different images of applicability on msgq.
It may also differ for others.  So I think it's important we first
establish a shared image of the intended usage of msgq with a highly
scalable operational environment. 

> > - "Multi-component support" and "Use of the bus from python-C++
> >   wrappers" were simply not easy to understand for me.  For a "short
> >   look", I didn't spend much time to try to understand them, so it's
> >   just an impression rather than specific points to improve.  It's
> >   just me, or some clarification help.
> 
> OK, I'll try to improve them.

Ah, okay, I think these examples clarify the intent.  Thanks.

---
JINMEI, Tatuya
Internet Systems Consortium, Inc.