BIND 10 #1899: performance issue of SQLite3 "iterate" query (w/ or w/o NSEC3)

[BIND 10 Development]

#1899: performance issue of SQLite3 "iterate" query (w/ or w/o NSEC3)
-------------------------------------+-------------------------------------
                   Reporter:         |                 Owner:  muks
  jinmei                             |                Status:  reviewing
                       Type:         |             Milestone:
  defect                             |  Sprint-20121009
                   Priority:         |            Resolution:
  medium                             |             Sensitive:  0
                  Component:  data   |           Sub-Project:  DNS
  source                             |  Estimated Difficulty:  6
                   Keywords:         |           Total Hours:  0
            Defect Severity:         |
  Medium                             |
Feature Depending on Ticket:         |
        Add Hours to Ticket:  0      |
                  Internal?:  0      |
-------------------------------------+-------------------------------------
Changes (by vorner):

 * owner:  vorner => muks


Comment:

 Hello

 First, I'd clarify why the result was sorted in the first place. We don't
 really need it sorted (I think the protocol does not mandate any order for
 xfrout, nor does when we insert stuff into the in-memory). We just need to
 collate the RRsets together. It is best done when we have a guarantee that
 all
 RRs of the same RRset are consecutive in the stream. And that is easier
 done by
 some kind of sorting. The fact that the NSEC3 namespace is sorted by a
 slightly
 different key is not really very interesting.

 When modifying the iteration back then to include NSEC3 namespace, I tried
 sorting each of the smaller queries together and then concatenating them
 together by union. But it might be that my knowledge of SQL is not good
 enough
 O:-). If you have an idea how to solve the above problem without sorting
 (and
 without accumulating the whole zone in memory), I'm all for it. But the
 extra
 work to make sure the NSEC3 things and the other RRs are sorted the exact
 same
 way is IMO not needed and only complicates the code.

 Also, the tests fail for me in the dbutil directory. I think you modified
 the
 schema only in some places, but not others:

 {{{
 5.2. Database is an old V1 database - upgrade
 2012-09-26 11:56:50.327 INFO  [b10-dbutil.dbutil] DBUTIL_FILE Database
 file:
 /home/vorner/work/bind10/bind10-devel-20120817/_build/src/bin/dbutil/tests/dbutil_test_tempfile_13574
 2012-09-26 11:56:50.327 INFO  [b10-dbutil.dbutil] DBUTIL_BACKUP created
 backup of
 /home/vorner/work/bind10/bind10-devel-20120817/_build/src/bin/dbutil/tests/dbutil_test_tempfile_13574
 in
 /home/vorner/work/bind10/bind10-devel-20120817/_build/src/bin/dbutil/tests/dbutil_test_tempfile_13574.backup
 2012-09-26 11:56:50.328 INFO  [b10-dbutil.dbutil] DBUTIL_UPGRADING
 upgrading database from V1.0 to V2.0
 2012-09-26 11:56:50.331 INFO  [b10-dbutil.dbutil] DBUTIL_UPGRADING
 upgrading database from V2.0 to V2.1
 2012-09-26 11:56:50.331 INFO  [b10-dbutil.dbutil] DBUTIL_UPGRADE_SUCCESFUL
 database upgrade successfully completed
 ERROR: upgraded schema not as expected
 *** FAIL
 }}}

 Thank you

-- 
Ticket URL: <http://bind10.isc.org/ticket/1899#comment:13>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development