[bind10-dev] MSGQ-NG requirements

[Michal 'vorner' Vaner]

Hello

On Mon, Apr 09, 2012 at 11:51:24PM -0700, JINMEI Tatuya / 神明達哉 wrote:
> Whether this (both speed and scale) matters depends on for what and
> how we want to use it.  Are we assuming we'll pass a list of the 1
> million zones with additional per-zone configuration via msgq?  What
> if the zones are primarily configured in a database backended data
> source?  What about ACLs, especially if they are defined per zone?

Well, if we couldn't pass our configuration and all this we pass through it now,
the msgq would be pretty useless. And if there are huge ACLs (multi-megabyte),
then it should be able to pass them.

I'd like to point out that parsing them in python might not be such a huge
problem (if the config manager can send them), because msgq does not need to
parse them. The routing header is separated from the payload and from the msgq
point of view, the payload could be anything, not a JSON, it does not need to be
inspected. I don't know if it is parsed now, but if it is, I don't think it is
correct.

> As for the speed, if we pass that amount of data over msgq with
> involving corresponding JSON parsing and validation, I'm afraid it
> could affect non-negligible impact on, e.g., the start up time.
> Partly due to these concerns and partly because of the (apparent) fact
> that the primary storage of a list of served zones is in a separate
> database system at least for database-based data sources, my general
> assumption was we'll only exchange higher level information over msgq
> such as a list of data sources used, thereby making the scalability
> concerns with msgq less critical.

Well, it depends. It could be convenient to pass many things through it. On one
side, it could simplify many things if we could just pass the DDNS packets from
auth to the module through it with file descriptors. On the other side, that
would probably be too slow, so it wouldn't work. And we may end up with some
receptionist thing anyway later, so it might not be worth worrying about anyway.
I don't know there.

> I'm not necessarily insisting that's the definite way to go, but
> apparently we are having different images of applicability on msgq.
> It may also differ for others.  So I think it's important we first
> establish a shared image of the intended usage of msgq with a highly
> scalable operational environment. 

I don't know. Maybe we don't want to pass the full list of zones, if they are in
a database, and we don't want to send the whole statistics, only a diff. But we
probably do want to send ACLs (where would the reside except for the config
manager?) and if the data source needs the list in its configuration (in-memory,
having each one loaded from some file or other data source), so I think we
should be able to send some large messages. I think we might get away not being
able to send tens of thousands of messages per second, on the other hand.

But anyway, this might be one thing to discuss during the F2F or some call, if
there's time.

With regards

-- 
Something is obviously wrong. The thing works.

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20120410/cdb78dcc/attachment.bin>