UpdateMgr asynch IO design for D2

[Thomas Markwalder]

On 5/10/13 7:30 AM, Stephen Morris wrote:
> On 08/05/13 21:32, Thomas Markwalder wrote:
>
>> The design is structured such that it can support both a
>> single-threaded operation as well as a multi-threaded operation.
>> The two modes of operation are described below.
> The question to be asked is whether, by keeping this flexibility, we are
> making compromises.

I'm not there is really a compromise.  The way I view it is that 
supporting multi-threaded
DNS updates is an extension via derivation.   The primary difference is 
that the DNS exchange
events are felt at the transaction level, versus the main thread level; 
and the NameTransaction hierarchy
would need a version that supports running in a thread.   It should be 
invisible to the upper layers,
other than a configuration switch.

As a stretch goal, I think it is fine.  In my mind it really doesn't 
alter the overall design.


>
>> Single-threaded operation: ========================== : while
>> (!shutdown) {
>>
>> // Execute all ready callbacks, or wait for one if
>> (io_service.poll() || io_service.run_one()) { : } }
> Use of io_service::poll() or io_service::run_one() within a while loop
> is possible, but it is not really the way that ASIO is designed be
> used.
I don't disagree, but one has to ask the obvious question.  Why are the 
methods
for poll, poll_one, and run_one?

> With ASIO, you call the run() method: this only returns when
> either there is no outstanding events or when an io_service::stop()
> call is made within one of the callback functions.  Otherwise,
> io_service::run() blocks until something is ready.  So the main
> function is effectively:
>
> main() {
>      Create io_service
>      Issue asynch read on configuration socket (callback: do_config())
>      Issue asynch read on update-request socket (callback: do_request())
>      io_service::run()
> }
>
> The the callback logic is something like:
>
> do_config() {
>      Apply configuration coange
>      Issue asynch read on configuration socket (callback: do_config())
>     }
> }
>
> do_request {
>     if (updates_in_progress >= max) {
>        Add request to queue
>     } else {
>        ++updates_in_progress
>        Post ddns_update_processing event
>     }
> }
>
> (The "updates_in_progress" counter is a way of restricting the number
> of pending I/Os.) The ddns_update_processing is the bit of the code
> that does the update and handles all the I/O with the nameserver, and
> wold look something like:
>
> ddns_update_processing() {
>     Do some processing
>     Issue I/O (callback: ddns_update_2)
> }
>
> ddns_update_2() {
>     Do some processing
>     Issue I/O (callback: ddns_update_3)
> }
>
> :
>
> (You are able to put all these callbacks as methods in a single object.)
>
> ddns_completion_function() {
>     if (no requests in queue) {
>       --updates_in_progress.
>     } else {
>         Remove request from the queue
>         Post ddns_update_processing event
>     }
> }
>
> It is the breaking of the processing into several functions that led
> to the stackless coroutines being used for the initial asynch I/O
> implementation in BIND 10.  With these, instead of breaking up a flow
> of processing into multiple functions, you can put it in one function.
>   A CORO_YIELD call is made at various points in that function,
> typically when issuing an I/O: that returns to the io_service::run()
> loop and provides the re-entry point when the I/O completes.
>
> The above logic allows the processing of multiple concurrent requests.
>   However, all this processing takes place within a single thread so is
> unable to take advantage of multi-core processors.

It may be completely possible to chain the callbacks appropriately such that
D2 handles everything it needs to in proper order.   If it becomes 
problematic,
the the poll||run_one structure will  still work.

>> Multi-threaded operation: ========================= :
>>
>> Each transaction, however, runs in its own thread, using its own
>> io_service object. Its events are never felt by the main thread.
>> It will wait for and process its own callbacks, driving itself
>> through its state machine.
> The question I have is what do we gain by using the ASIO model within
> each thread?  ASIO is of benefit when something else in the same
> thread can do something while the I/O is pending.  If the thread is
> doing nothing in the meantime, then it might as well do synchronous I/O.

Agreed, they become effectively synchronous.  What you do gain, is that 
moving
between single and multi threading is largely a matter of which 
io_service object
a transaction uses.

> ASIO would still be used to handle reads of DDNS requests and reads of
> configuration updates.  So the main loop would be as described above.
>   But with multi-threaded operation the queue is protected by a mutex;
> a a condition variable is also present for threads to wait on.  When a
> request is added to the queue, threads waiting on the condition
> variable are notified.  Their logic is:
>
> do_processing() {
>     while (true) {
>        do {
>           Wait on condition variable
>           Attempt to get request from queue
>        } while (we don't have a request)
>
>        Process request using synchronous I/O
>     }
> }
>
> Apart from the use of a mutex over getting requests from the queue,
> the threads have zero interaction with one another.
>
>
> Finally, a point we should note is that the BIND 10 philosophy has
> been for a single-threaded multi-process approach.  The belief here is
> that it is easier to debug and is more robust.  We could do the same
> for the DDNS process by either using shared memory or with a receptionist.
>
> Shared memory: each of the single-threaded processes listens for the
> requests on the same socket and uses a queue shared between processes
> to hold pending requests.  Sharing the socket ensures that even if a
> single process crashes, requests still get services.  A shared queue
> means that the optimisation logic we plan to add that requires
> scanning the queue will work whatever process picks up the request.
>
> Receptionist: a single process that reads requests from the server and
> adds them to the queue.  The receptionist farms them out to the worker
> processes after doing any queue optimisation logic requested.
Based on your earlier comment:

"We have talked about requiring asynchronous operation. We can either go 
for: a) A multi-threaded approach. b) A single-threaded approach. The 
former appears to be going away from the original BIND 10 
single-threaded process approach (although I think at least one BIND 10 
process is now multi-threaded)."

I hadn't considered a multi-process solution.  If this is the preferred 
direction then I'll need to adjust for it.

>
> Stephen