[bind10-dev] "Percentage done", was Re: questions about task 88 and 89

[Shane Kerr]

Jeremy,

[ ignoring the largely correct clarification you sent ]

On Thu, 2010-05-06 at 05:22 -0500, Jeremy C. Reed wrote:
> "%done" is for periodically showing the percentage of the master file 
> data that is loaded until complete. I am not sure how relevant or easy 
> that is to do now as I don't know if it pre-parses the loaded data to 
> get something to compare with. It is probably not required in the first 
> "verbose" option.

The basic idea is to have some idea of how long one has to wait. This
can be done by doing something that looks a bit like:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "xn--fsqu00a.cn"
5342 Resource Record(s) loaded in 2 second(s) (0.5% of file)

After a while you would see something like:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "xn--fsqu00a.cn"
991337 Resource Record(s) loaded in 456 second(s) (91.3% of file)

And finally you would see:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "xn--fsqu00a.cn"
1000000 Resource Record(s) loaded in 463 second(s) (100.0% of file)
Committing changes... done.
$

We can use os.isatty(sys.stdout.fileno()) on stdout to detect whether we
are going to a interactive session, so we don't spam people with updates
who are using this from scripts.

Minimal refactoring would then be to change the load() function in
src/lib/python/isc/datasrc/sqlite3_ds.py to include a callback function
and callback data parameter which would be used after every record is
complete. Maybe some sort of state can also be passed, so that we can
output progress like "cleaning old data" as well.

The callback function can use "\r" to move the cursor back to the
beginning of the line, to allow updates that don't scroll madly. It can
keep a counter so we don't output on every record. I would guess
something like every 100 records or so one would check the time, and
output if more than a half second or so of clock time had passed.

We can use zonefile.tell() to get the position in the file, and
os.fstat() to find the total file size, and use this as an estimate as
to our progress through the file. It's not perfect but should be enough
to give users a useful idea of progress.

I don't think it's *that* much work, and it is the kind of thing that
makes me happy as a user when I see it. Otherwise I end up using
"strace" or "lsof" and trying to follow progress of large operations
through system level methods, which is less than ideal. :)

--
Shane