BIND 10 #2855: introduce a "zone builder" thread in memory manager

[BIND 10 Development]

#2855: introduce a "zone builder" thread in memory manager
-------------------------------------+-------------------------------------
            Reporter:  jinmei        |                        Owner:
                Type:  task          |                       Status:  new
            Priority:  medium        |                    Milestone:
           Component:  shmem         |  Sprint-20130625
  manager                            |                   Resolution:
            Keywords:                |                 CVSS Scoring:
           Sensitive:  0             |              Defect Severity:  N/A
         Sub-Project:  DNS           |  Feature Depending on Ticket:
Estimated Difficulty:  4             |  shared memory data source
         Total Hours:  0             |          Add Hours to Ticket:  0
                                     |                    Internal?:  0
-------------------------------------+-------------------------------------
Description changed by jinmei:

Old description:

> Subtask of #2830.  Depend on #2854.
>
> Like b10-auth, we'll need to be able to load zones into memory in
> "background"; memmgr doesn't have to answer queries, but it should
> still need to handle commands from other modules while it's loading
> zones.
> There could be several possible ways, but we should probably use a
> separate thread like b10-auth.
>
> Assuming that approach, in this task we introduce this thread and
> implement some communication channel between the main thread.  Right
> now the "builder thread" doesn't do anything, except probably a
> "shutdown" operation.
>
> Consider using some wrapper of thread so it can be tested without
> actually spawning threads.

New description:

 Subtask of #2830.  Depend on #2854.

 Like b10-auth, we'll need to be able to load zones into memory in
 "background"; memmgr doesn't have to answer queries, but it should
 still need to handle commands from other modules while it's loading zones.
 There could be several possible ways, but we should probably use a
 separate thread like b10-auth.

 Assuming that approach, in this task we introduce this thread and
 implement some communication channel between the main thread.  Right
 now the "builder thread" doesn't do anything, except probably a
 "shutdown" operation.

 Consider using some wrapper of thread so it can be tested without
 actually spawning threads.

 Some more specifics based on #2854 (although its details may change
 after review):

 - First, I think sub-thread is still handy than sub-process; while
   threading can cause a nightmare and some of the existing use of it
   within BIND 10 is really poor, we shouldn't be too cautious about
   threads.  I'd leave the final decision to the actual implementor,
   but consider how it works for both the details of this task and
   #2856 (maybe should confirm it actually works with experimental code
   for #2856).
 - We'll need to make some extensions to the
   server_common.bind10_server module:
   - Add "watch_fileno" method to `BIND10Server`:
 {{{#!python
     def watch_fileno(self, fileno, rcallback, wcallback, wcallback):
         """Register the fileno for the internal select() call.

         *callback's are callable objects which would be called when
         read, write, error events occur on the specified fileno.
         """
 }}}
   - Add "_shutdown_module()" method it will be called from
     `BIND10Server._run_internal()` for module-specific shutdown
     processing:
 {{{#!python
     def _run_internal(self):
         ...
         self._shutdown_module()  # <= added
         self._mod_cc.send_stopping()
 }}}
 - In `Memmgr._setup_module()`, create the builder thread:
   - create a socketpair.  this will be used by the builder to notify
     the main thread of some updates.
   - register the socket with watch_fileno().
   - create a `threading.Condition` object.  this will be used with a
     list (queue) to exchange commands from the main thread to the builder.
   - create a lock object. this will be used with another list (queue) to
     exchange command results (and other updates) from the builder to the
     main thread.
   - spawn a thread passing the socketpair, lists, the `Condition`
     and lock objects.
 - Add `Memmgr._shutdown_module()`.  It sends a shutdown command to
   the builder thread, then joins it (with other cleanups).
 - Add a builder module in the isc.memmgr package:
   - the builder is basically waiting on the command queue for commands
     from the main thread:
 {{{#!cpp
 class MemorySegmentBuilder:
     def run(self):
         while True:
             self.__cv.acquire()
             while len(self.__command_queue) == 0:
                 self.__cv.wait()
             # move the queue content to a local queue
             self.__cv.release()

             # run commands in the queue in the given order.  For now, it
             # only supports the "shutdown" command, which just exits from
             # the main loop
 }}}

--

-- 
Ticket URL: <http://bind10.isc.org/ticket/2855#comment:6>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development