[bind10-dev] Low-bandwidth messaging API design

[Michael Graff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been coming up with what I think are requirements for such a
system.  I did this with some assumptions on my end:

(1)  What a "large transfer" means is different to each user of such an
API, and in fact we should not define what "large" is here other than to
say "if you send something that is 1 MB, it might work, but it may be
rather slow."

(2)  We may want to broadcast to many different receivers at once.

(3)  I don't think we want senders and receivers to need to know about
one another in advance.  That is, something might want to say "zone
isc.org has changed from serial 1234 to serial 1235" and have things
that care about this just pick up the fact.  A sender should not need to
know who is receiving this information, and a receiver should not need
to know who is going to be sending, just where.

(4)  It has to scale to many machines eventually, and efficiently.

(5)  It has to be simple to use.

(6)  It is inherently unreliable; to introduce reliability into the mix,
we will add an API on top of this one that performs such.  I will design
that one as well using primitives from the unreliable one.

(7)  It is "snoopable" in that it will be easy to write something to
listen in on conversations between senders and receivers, perhaps ALL
senders and receivers on a machine.

So, here are the initial requirements:

(1)  Supports small messages efficiently, where small is anything less
than 20k.  Efficiently simply means without memory bloat or excessive
latency.  Larger messages should be supported, up to about 1 MB.

(2)  Clients of this API will subscribe to events they care about, which
occur in "groups."  An event is a message notification; it may be sent
to a specific recipient, or to any of a number of different recipients
within a group.  The messaging system handles this distribution, the
client does not.

(3)  We need some sort of service discovery system which will allow
learning of new services as they come online, and for clients to
announce their status to accept load/shutting down, etc.

(4)  This must be easy and unobtrusive to use.

(5)  Binary as well as ASCII data in various formats must be efficient
to use.

Prior work:

dbus:  Does not do multi-machine, but is pretty simple to use, and is a
standard of sorts.  There was mention of a multi-machine system many
years ago, but I have yet to see it materialize.

XMPP:  AKA Jabber.  This might well work for a transport; it supports
multiple recipients (to a certain degree) as well as single targets.

openreg:  This might satisfy most of what we want, and has directly
influenced my thought process on what we'll want and how we'll want to
use it.

I'll document how openreg's communication system worked in another
message.  I am leaning somewhat on the XMPP solution for a number of
reasons, not the least of which is we may not have to write a server to
handle these messages.

- --Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpbkvsACgkQ+NNi0s9NRJ1sIACfUDUVJ9VIKwKi2Rap8RyIxo/v
MZgAmwdEMblQO9TF/3vjc5O1OAoO6GnT
=zufy
-----END PGP SIGNATURE-----