[bind10-dev] auth server benchmarks and resource usage

[Stephen Morris]

On 23 Nov 2010, at 17:47, Jeremy C. Reed wrote:

> On Tue, 23 Nov 2010, Jerry Scharf wrote:
> 
>> Why not build the database once and just restart the server using this
>> database and test queries and caching against it when you want to?
> 
> Yes, that is what I am doing. I did this last week. It took around 7 
> hours to build. It contains 100 million records:
> 
> [jreed at bind10-testing1 dnsbenchsuite]$ du -h ./tests/hugetld/master.zone.file ./tests/hugetld/zone.sqlite3
> 2.6G    ./tests/hugetld/master.zone.file
> 15G     ./tests/hugetld/zone.sqlite3
> 
> But I need to do again since the b10-loadzone tool is broken and parsed 
> my master file wrong. (I opened a ticket for this.) Also b10-auth is 
> basically unusable with large database. (I opened ticket for this too.)

The following article is interesting and suggests a number of ways in which the bulk insert performance of SQLITE can be improved: http://stackoverflow.com/questions/1711631

One suggestion that could be tried without the need to change b10-loadzone is to drop the indexes on the records table, loading the data, then recreate the indexes.  Other suggestions (such as setting the "synchronous" pragma to "off") would probably need to be incorporated into the code.

Incidentally, the "synchronous" pragma idea does suggestion that b10-loadzone could operate in different modes.  The pragma controls the operation of writing to disk; by default sqlite3 will pause at critical moments to ensure that data is written to disk before continuing.  If there is a failure/operating system crash at a critical moment the database is uncorrupted.  With the pragma set to "off", it passes the data to the operating system and continues writing; a failure could cause the database to become corrupted.

If I already have a large record database and am loading some more records in the course of normal operation, I would probably want to run in "safe" mode.  However, if I'm setting up the system and am loading millions of records, I don't care if the database is corrupted on a crash - I'll just restart the operation.  So could b10-loadzone run by default in slow (but safe) mode but have a flag that lets it operate more quickly (but not quite as safe)?

Stephen