BIND 10 #2219: switch to memory-efficient version of in memory data source

[BIND 10 Development]

#2219: switch to memory-efficient version of in memory data source
-------------------------------------+-------------------------------------
                   Reporter:         |                 Owner:  jinmei
  jinmei                             |                Status:  accepted
                       Type:  task   |             Milestone:
                   Priority:         |  Sprint-20120925
  medium                             |            Resolution:
                  Component:  data   |             Sensitive:  0
  source                             |           Sub-Project:  DNS
                   Keywords:         |  Estimated Difficulty:  6
            Defect Severity:  N/A    |           Total Hours:  0
Feature Depending on Ticket:         |
  scalable inmemory                  |
        Add Hours to Ticket:  0      |
                  Internal?:  0      |
-------------------------------------+-------------------------------------

Comment (by jinmei):

 trace2219 is ready for review.

 It's developed on top of #2218, and cannot be merged until #2218 is
 done, but changes are basically independent, so it should be
 reviewable without waiting for the completion of #2218.

 This branch consists of a relatively straightforward adjustment for
 the users of the in-memory data source with several small fixes to
 issues I encountered.

 Changes that may need explanation would be:

 - First two commits (up to 40fe836) are dependencies and should be
   ignored.
 - 8d09311 is the main change.  I needed to made non-trivial changes to
   the client list unittests, but the changes to the main code should
   be pretty straightforward.
 - e1ae684 is an important fix to the new in-memory finder.  without
   this non DNSSEC queries won't be handled correctly.
 - 1b35c2a is another fix to the zone finder.  this is relatively
   minor, but needed to cover some corner cases.
 - 56df3cb might be a bit tricky.  I think we should rather hide
   details like `DataSourceInfo` from applications, but for the purpose
   of this migration I wanted to minimize the size of the change.
 - 5261f93 is another tricky workaround.  maybe we should think about
   cleaner solution, but I think it's okay for now.
 - b4913f7 is an unrelated bug fix, but necessary for the new
   implementation to work correctly.  the fix should be obvious.
 - a61af89 is not absolutely necessary for the purpose of this
   migration, but without this usability will be affected for huge
   zones (which is the main purpose of the migration).

 And, there's one remaining part that still uses the old
 implementation: the static data source (client).  To switch it to the
 new one, we'd need some wrapper `DataSourceClient` derived class
 because it would need a separate memory segment.  The change shouldn't
 be difficult, but since the branch is getting bigger and
 memory-efficiency shouldn't be an issue for the static case anyway, I
 thought it'd be better to defer it so we can complete and merge the
 important part sooner.  I'll create a separate ticket for this.

 Final note: some system tests fail due to bugs that existed at the
 point of branching.  I confirmed that they pass with merging the
 latest master.

 This is the proposed changelog entry:
 {{{
 477.?   [func]          team
         Memory footprint of the in-memory data source has been
         substantially improved.  For example, b10-auth now requires much
         less memory than BIND 9 named for loading and serving the same
         zone in-memory.  This is a transparent change in terms of user
         operation; there's no need to update or change the configuration
         to enable this feature.
         Notes: multiple instances of b10-auth still make separate copies
         of the memory image.  Also, loading zones in memory still suspends
         query processing, so manual reloading or reloading after incoming
         transfer may cause service disruption for huge zones.
         (Multiple Trac tickets)
 }}}

-- 
Ticket URL: <http://bind10.isc.org/ticket/2219#comment:5>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development