Fwd: UpdateMgr asynch IO design for D2

[Tomek Mrugalski]

See my comments inline.

On 09.05.2013 18:43, Thomas Markwalder wrote:
> Stephen asked that I forward this to the bind10 team.s
> 
> Introduction: =============
> 
> This is the overall design for D2 and how it will handle its
> asynchronous, event driven nature.
> 
> Background: ===========
> 
> A review of the asynchronous events that D2 must handle yields the
>  following list:
> 
> 1. Name change request messages from DHCP servers and/or a
> house-keeper process. When messages are received: a. They need to
> be added to the queue as NameChangeRequests. b. D2 must be made
> aware that the Queue has entries.
We haven't started designing the house-keeper process. In a normal
deployment I think it will be a separate process, but in some
deployments (CPE?) it could be just a function called from within the
single dhcp server process (just as isc-dhcp or dibbler do). I think
it would be good to implement house-keeper routines as a single
function that is called periodically. When house-keeper runs in
stand-alone mode, it is just called every configured interval (e.g.
once every second). When configured to run from within dhcp server,
the server can track lease expiration and call house-keeper at the
specified timeout (or call select() for some configured period and
call house-keeper function appropriately).

We should also need to keep in mind that while we have now a single
dhcpv4 and single dhcpv6 process, we have vague plans to support
multi-cores.

The implication for DDNS is that while the simple case is just one
producer, there potentially may be many producers (multiple instances
of dhcpv4, multiple instances of dhcpv6, v4 house-keeper, v6
house-keeper) for a single consumer (D2).

> 2. DNS message exchange events from DNS update exchanges with DNS
> servers. Each transaction can consist of multiple sends and
> receives. Each send and receive outcome amounts to an event.  These
> events are used to move each transaction through its state machine.
> At points in which the transaction must wait for IO, control should
> be yielded.
> 
> 3. Control Events - primarily "shutdown" but there could be
> others?
Right now dhcpv{4,6} support 2 control events: configuration update
and shutdown command. I'm not sure how we'll decide to store keys. If
we keep everything in bind10 configuration, then we're ok. If they are
kept in external files, perhaps some form or "re-read keys" command
would be needed, when keys change?

> 4. Completed Transactions - While not an event per se, D2 does need
> to monitor its list of transactions, for those that have
> completed.
Yes. For logging purposes and possibly some other events, like hooks.

> The design should reuse existing libraries where possible and to
> that end the DNS UPDATE exchanges will use the asiolink library to
> carry out socket IO.
Ok. In that case you should take a closer look at src/bin/auth. It is
using asiolink for processing DNS queries, configuration updates and
control commands. When I was writing b10-dhcp{4,6} modules, I roughly
based implementation on auth, but had to do some tricks to also poll
control socket. You won't need some things that are done in
ControlledDhcv6Srv::establishSession() See the part of code that sets
handlers in strange way, then gets control socket and sets using
set_session_socket(). You most probably don't need any of these.

> In addition, the NameChangeIPC will be socket-based rather than
> use boost::interprocess::messagequeue. This will allow the use of
> the asiolink library for handling the receipt of name change
> request messages as well. This unifies the asynchronouse IO and
> event callback mechanisms for the two external event sources.
Will this be like a UDP socket open on loopback? Or do you think about
unix-socket?

> The design is structured such that it can support both a
> single-threaded operation as well as a multi-threaded operation.
> The two modes of operation are described below.
> 
> 
> Single-threaded operation: ==========================
> 
> In single-threaded operation, the main processing loop performs a
> single "blocking" wait for events from all the event sources.
You mentioned that the code will use asiolink, so will be a

> Since both the NameChangeIPC and DNS exchanges will use
> asiolink/socket IO, D2 can use a single asiolink::io_service object
> to wait for and execute callbacks for both event sources.
Ok.

> The QueueMgr's NameChangeIPC callback will create NCRs from the
> packets received and add them to the NCR Queue.
> 
> For a DNS transaction, its DNS exchange callback(s) will "post" the
>  appropriate event to transaction's state machine. This drives the
> state machine forward until next IO-wait is reached or the state
> machine completes.
Auth is implemented in a way that calls io_service.run() and it
returns only when shutting down the server. D2 can use asiolink
differently, if needed. For me, the control flow was kind of strange
for DHCP server, so I chose select() approach instead. It may be more
appropriate for D2.

> 
> The D2::UpdateMgr main run loop will look something like this:
> 
> while (!shutdown) {
> 
> // Execute all ready callbacks, or wait for one if
> (io_service.poll() || io_service.run_one()) {
> 
> // One or more of the following has occurred //   a. NCR message
> received and has been queued //   b. Pending DNS IO has completed 
> //   c. Interval timer expired
> 
> if (QueueMgr->queueNotEmpty()) { // Dequeue next pending request 
> NCR = QueueMgr->dequeue();
> 
> // Start a new transaction for the NCR 
> instantiate_transaction(io_service, NCR); }
> 
> handle_finished_transactions();
> 
> } else {
> 
> // we are here because there is no more work to do // somebody
> stopped io_service, shutdown request likely culprit
> 
> } }
I think it has one major issue. If there is nothing to process (poll()
returns immediately), the code will loop like crazy and will eat 100%
cpu, right?


> Multi-threaded operation: =========================
> 
> As with the single-threaded operation, D2 will wait for and execute
>  callbacks for the NameChangeIPC.  QueueMgr runs in the main D2
> thread, its NameChangeIPC callback will create NCRs from the
> packets received and add them to its NCR Queue.
> 
> Each transaction, however, runs in its own thread, using its own
> io_service object. Its events are never felt by the main thread.
> It will wait for and process its own callbacks, driving itself
> through its state machine.
> 
> The D2::UpdateMgr main run loop remains effectively the same as
> before:
> 
> while (!shutdown) { // Execute all ready callbacks, or wait for
> one if (io_service.poll() || io_service.run_one()) { // One or more
> of the following has occurred //   a. NCR message received and has
> been queued //   b. Interval timer expired
> 
> if (QueueMgr->queueNotEmpty()) { // Dequeue next pending request 
> NCR = QueueMgr->dequeue();
> 
> // Start a new threaded transaction for the NCR instantiate
> threaded_transaction(NCR); }
> 
> handle_finished_transactions(); } else { // we are here because
> there is no more work to do // somebody stopped io_service,
> shutdown request likely culprit } }
> 
> Again, to ensure we periodically sweep, an Interval timer will be
> used to generate a ready callback at regular intervals.
> 
> The only real difference between the two loop's is in the type of 
> transaction instantiated.
I remember that at one point in time, we had plans for doing simple
prototypes for chosen approach to see if the scale as we think they
scale. We did similar thing before we chose MySQL for DHCP backend.
The result - set of 3 micro-benchmarks are available in
tests/tools/dhcp-ubench. I know that our current schedule is...
challenging, so I understand if we choose to not do it.

Do we plan to implement both approached by August? I think
implementing single-threaded approach will be sufficient for now.
multi-threaded approach could be considered a stretch goal.

In general, good approach. You need to solve the 100% CPU usage issue
and it's ready for implementation.

Tomek