[bind10-dev] "Percentage done", was Re: questions about task 88 and 89

[Shane Kerr]

Likun,

On Wed, 2010-05-12 at 10:28 +0800, zhanglikun wrote:
> > [ ignoring the largely correct clarification you sent ]
> > 
> > On Thu, 2010-05-06 at 05:22 -0500, Jeremy C. Reed wrote:
> > > "%done" is for periodically showing the percentage of the master file
> > > data that is loaded until complete. I am not sure how relevant or easy
> > > that is to do now as I don't know if it pre-parses the loaded data to
> > > get something to compare with. It is probably not required in the first
> > > "verbose" option.
> > 
> > The basic idea is to have some idea of how long one has to wait. This
> > can be done by doing something that looks a bit like:
> > 
> > $ b10-loadzone example.db
> > Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
> > Zone name is "xn--fsqu00a.cn"
> > 5342 Resource Record(s) loaded in 2 second(s) (0.5% of file)
> > 
> 
> How b10-loadzone display percentage done for included zone file?
> 
> if zone file 'cn.zone' includes 'sub.cn.zone', and zone file 'sub.cn.zone'
> includes
> 'sub.sub.cn.zone', 

Ah, right! :-P

> My suggestion is :
> 
> When loading records in file 'cn.zone', just report the percentage of zone
> file 'cn.zone', like: 
> 5 Resource Record(s) loaded in 456 second(s) (1% of file 'cn.zone')
> 
> Then, when loading records in included file, add percentage report for the
> included zone file, like:
> 10 Resource Record(s) loaded in 456 second(s) (10% of file 'cn.zone'， 15%
> of included file 'sub.cn.zone')
> 
> seconds later would display like:
> 345 Resource Record(s) loaded in 456 second(s) (10% of file 'cn.zone'， 90%
> of included file 'sub.cn.zone')
> 
> seconds later would display like:
> 2664 Resource Record(s) loaded in 456 second(s) (10% of file 'cn.zone'， 40%
> of included file 'sub.sub.cn.zone')
> 
> seconds later would display like:
> 4664 Resource Record(s) loaded in 456 second(s) (10% of file 'cn.zone'， 80%
> of included file 'sub.sub.cn.zone')    
> 
> seconds later would display like:
> 25444 Resource Record(s) loaded in 456 second(s) (15% of file 'cn.zone')
> 
> Any other suggestions?

Hm... tricky.

I think this is basically the right idea, but... we have some small
problem when output lines get too long. If you have a terminal set up
with 80 columns, then you can see this:

shane at shane-asus-laptop:~$ perl -e 'print "X"x80,"\r","---\n"'
---XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
shane at shane-asus-laptop:~$ perl -e 'print "X"x81,"\r","---\n"'
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
---

Notice how in the first case we get the "---" at the beginning of the
line, where we want it. What this means is that if we output progress
and it is longer than the line we are on, it will come out as new lines
instead of on the same line.


What I propose is to use your technique, but also to change the output
if we need to fit it on the display. We would just be less verbose, and
remove content if we are too wide. So in the normal inclusion, we might
have:

4664 RR loaded in 456s (10% of 'cn.zone', 40% of 'sub.cn.zone')

For multiple sub-zones, we might have:

4664 RR loaded in 456s (10% of 'cn.zone', ..., 80% of 'sub.sub.cn.zone')


Another possibility is to only ever output status of the current file.
So, we start normally:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
Loading from file "cn.zone"
5342 Resource Record(s) loaded in 2 second(s) (0.5% of file)

If an included file is encountered, we then let the user know that we
have started loading this:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
Loading from file "cn.zone"
Including file "sub.cn.zone" from "cn.zone"
16773 Resource Record(s) loaded in 45 second(s) (80.2% of included file)

Likewise for multiple layers of inclusion:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
File is "cn.zone"
Including file "sub.cn.zone" 
Including file "sub.sub.sn.zone" 
45315 Resource Record(s) loaded in 55 second(s) (40.0% of included file)

When an included file finishes we report this and continue:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
File is "cn.zone"
Including file "sub.cn.zone" 
Including file "sub.sub.cn.zone"
Included file "sub.sub.cn.zone" done
47341 Resource Record(s) loaded in 59 second(s) (85.1% of included file)

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
File is "cn.zone"
Including file "sub.cn.zone" 
Including file "sub.sub.cn.zone"
Included file "sub.sub.cn.zone" done
Included file "sub.cn.zone" done
47341 Resource Record(s) loaded in 59 second(s) (1.2% of file)

And finally we finish:

$ b10-loadzone example.db
Using SQLite3 database file "/var/lib/bind10/test-sqlite.db"
Zone name is "cn"
File is "cn.zone"
Including file "sub.cn.zone" 
Including file "sub.sub.cn.zone"
Included file "sub.sub.cn.zone" done
Included file "sub.cn.zone" done
File "cn.zone" done
14235212 Resource Record(s) loaded in 512 second(s)
$


I really don't have any intuition about which is the better solution.
The first one is possibly a bit sexier, but also includes less
information. It will be harder to code, although that may be fun. It may
ultimately be less confusing to the user because of this, while at the
same time making debugging harder. :-/


*** WARNING: Unnecessary Detail Below ***

Now, I am definitely polishing here, but we can find the width of the
screen and use that to change our output. Here's some code that seems to
work fairly well (look for the getTerminalSize() function):

http://stackoverflow.com/questions/566746/how-to-get-console-window-width-in-python

Note that if we were *really* polishing we could check the size before
every time we write, something like this:

class loadStatusTracker:
    def __init__(self, file_name, file_handle, parent=None):
        self.record_count = 0
        self.last_display_record_count = 0
        self.last_display_time = 0
        self.file_name = file_name
        self.file_size = os.fstat(file_handle.fileno()).st_size
        # when we start an included file, we need to know the 
        # status of file this is included *from* for the display
        self.parent = parent

    def record_added(self, load_status):
        self.record_count = self.record_count + 1
        # arbitrary values
        if (self.record_count - self.last_display_record_count) > 100:
            if (time.time() - self.last_display_time) > 0.5:
                self._display(load_status)

    def _display(self, load_status):
        (height, width) = getTerminalSize()
        # logic to properly format our display would go here...

--
Shane