[bind10-dev] Low-bandwidth messaging API design

[Shane Kerr]

Michael,

Thanks for this! Apologies for taking so long to get back, but I have
been underground (literally). 

On Mon, 2009-07-13 at 15:03 -0500, Michael Graff wrote:
> I've been coming up with what I think are requirements for such a
> system.  I did this with some assumptions on my end:
> 
> (1)  What a "large transfer" means is different to each user of such an
> API, and in fact we should not define what "large" is here other than to
> say "if you send something that is 1 MB, it might work, but it may be
> rather slow."

Makes sense. "rather slow" being on the order of seconds, right?

> (2)  We may want to broadcast to many different receivers at once.

Definitely.

> (3)  I don't think we want senders and receivers to need to know about
> one another in advance.  That is, something might want to say "zone
> isc.org has changed from serial 1234 to serial 1235" and have things
> that care about this just pick up the fact.  A sender should not need to
> know who is receiving this information, and a receiver should not need
> to know who is going to be sending, just where.

Agreed.

We do need unicast as well as broadcast communication. For instance,
when a component starts up, it may need to query the status of other
components. These answers probably should only be sent to the newly
started component. (I see that this fits in your requirements below, now
that I look ahead...)

> (4)  It has to scale to many machines eventually, and efficiently.

"Efficiently" meaning not a lot of extra... anything. (Packets, memory,
delay, CPU use, and so on.)

> (5)  It has to be simple to use.

Yup.

> (6)  It is inherently unreliable; to introduce reliability into the mix,
> we will add an API on top of this one that performs such.  I will design
> that one as well using primitives from the unreliable one.

I think we need to be careful about this. I am quite okay allowing
dropped packets because of limited buffer space, but I'd prefer we use
TCP rather than UDP for inter-machine messaging (for example).

> (7)  It is "snoopable" in that it will be easy to write something to
> listen in on conversations between senders and receivers, perhaps ALL
> senders and receivers on a machine.

Agreed.

> So, here are the initial requirements:
> 
> (1)  Supports small messages efficiently, where small is anything less
> than 20k.  Efficiently simply means without memory bloat or excessive
> latency.  Larger messages should be supported, up to about 1 MB.
> 
> (2)  Clients of this API will subscribe to events they care about, which
> occur in "groups."  An event is a message notification; it may be sent
> to a specific recipient, or to any of a number of different recipients
> within a group.  The messaging system handles this distribution, the
> client does not.

What about security? Do we need to restrict subscription in some way?

> (3)  We need some sort of service discovery system which will allow
> learning of new services as they come online, and for clients to
> announce their status to accept load/shutting down, etc.
> 
> (4)  This must be easy and unobtrusive to use.
> 
> (5)  Binary as well as ASCII data in various formats must be efficient
> to use.

This looks good. Language and portability requirements are not written,
but I guess we assume those.

> Prior work:
> 
> dbus:  Does not do multi-machine, but is pretty simple to use, and is a
> standard of sorts.  There was mention of a multi-machine system many
> years ago, but I have yet to see it materialize.
> 
> XMPP:  AKA Jabber.  This might well work for a transport; it supports
> multiple recipients (to a certain degree) as well as single targets.
> 
> openreg:  This might satisfy most of what we want, and has directly
> influenced my thought process on what we'll want and how we'll want to
> use it.
> 
> I'll document how openreg's communication system worked in another
> message.  I am leaning somewhat on the XMPP solution for a number of
> reasons, not the least of which is we may not have to write a server to
> handle these messages.

We'll need a layer on top of XMPP of course.

Onward to the OpenReg mails... :)

--
Shane