[bind10-dev] Comment to Statistics Ideas

[fujiwara at wide.ad.jp]

Shane-san,

Current my idea of Statistics
   handle minimal set of static couters which are compatible with BIND 9.7.
     from bind-9.7.0rc1/bin/named/statschannel.c:init_desc()
   have extensibility (each zone couters, ...)
   uses SQLite as internal database.
     HTTP/SNMP daemon share the database.
     HTTP interface outputs XML statistics compatible with BIND 9.7.
   sends some command to all modules via CCchannel.
     get*Stat*
     set*Stat*: specify couters list

> From: Shane Kerr <shane at isc.org>
>> * My idea was
>>  - Stats daemon sends a collecting command to each AuthSrv via the CC channel
>>     periodically
>>  - Stats daemon totals each counter collected from AuthSrvs
>>  - Stats daemon publishes one counter set
>>  - Stats daeomn holds short time data only
>>  - SNMP Stats server respond the recent couter set
> 
> So, the main purpose of the stats daemon is to collect counters from a
> number of identical daemons and combine them. Do you think this would be
> a general-purpose thing, or specific to AuthSrv counters?

I didn't define it now.

First version of Stat daemon collects specific AuthSrv couters  (Minimal sets),
I think.

>> What we need to define and fix:
>> 
>>  - Stats couter list
>>    - static or dynamic?
>>    - if dynamic, need to consider counter managing command.
> 
> I tend towards dynamic, as it allows us flexibility for future changes,
> and avoids versioning issues (matching the stats daemon with specific
> AuthSrv versions).

Minimal sets of AuthSrv couters + Dynamic sets

>>  - Statistics collecting command/protocol in CC channel
>>    - data format
> 
> Simple XML makes sense, unless the statistics become huge. Maybe there
> should be a way to ask for changes only, in order to minimize the
> overhead?

I considered one idea. Is it correct?
  { "bind97compat": {
      "requestv4": number,
      "requestv6": number,
      ...
    }
    "extended": {
      "zone_couters": {
        "jp": number,
        "org": number,
        ...
      }
      "other_counters": {
        "memory_usage": ...,
        "socket_usage": ...,
      }
    }
  }

> So, something like:
> 
>   Command:   getAllStats
>   Arguments: NONE

I prefer Stat Daemon sends stat_id and respondent copies it in its response.

>   Returns:   statistics, stat_id
>   
>         Returns all statistics from a server, and a statistics
>         identifier that can be used to retrieve stats efficiently in the
>         future.
> 
>   Command:   getNewStats
>   Arguments: stat_id
>   Returns:   statistics, stat_id
> 
>         Returns all statistics from a server that have been modified
>         since the given stat_id, as well as a statistics identifier that
>         can be used to retrieve stats efficiently in the future.

This case, respondent module implementation may be complicated, I
think.  Most of simple couters may change. The effectivity of modified
check may be limited.

-snip-

>>  - Collecting frequency
> 
> The simplest thing is to have a configuration parameter and let the
> administrator set this. We probably want to use a fractional number of
> seconds, since some people may want this updated more than once per
> second (crazy but maybe nice for GUIs).
> 
> This may be something that various stats reporting programs want to
> modify. So, for example, a BIND 10 system may dump stats to a log file
> every hour, but someone may start SNMP monitoring during a problem and
> want these every 30 seconds.
> 
> Another possibility is that these are collected "on demand". This may be
> useful for XML where we want stats, but if someone isn't asking then we
> don't need them updated. Of course, this may cause a bit of latency when
> getting stats via XML, but I think this should be quite minimal.

I prefer default frequency is every 10 or 15 seconds.

>>  - How to publish
>>    - SNMP
>>    - XML ?
> 
> We have to have XML that is compatible with the BIND 9 XML stats (only
> better). ;)

Ok.

>>    - command?
> 
> Probably useful. I'm not sure whether it makes sense to go through the
> command & control daemon or use the XML daemon for this.
> 
>>  - How to implement internal database.
> 
> I suggest SQLite is probably the most straightforward. (See below for
> more...)

I will check/learn SQLite.

>> After here, comment to StatisticsIdeas in-line.
>> -------------------------------------------------------------------------------
>> | = Statistics Ideas =
>> | 
>> | This wikipage is for brainstorming ideas for the statistics interface.
>> | 
>> | == Existing BIND 9 features ==
>> | 
>> |   * have around 130 counters, such as IPv6 requests received, Requests with TSIG received, Zone transfer requests rejected, etc.
>> |   * almost all statistics counters supported in BIND 8
>> 
>> Need to define which counters each AuthSrv has.
>> 
>> |   * XML-based statistics interface
>> 
>> Is the interface outputs totals of all AuthSrvs' counters?
> 
> Yes, I think so.
> 
>> |   * statistics counters about internal status (sockets/tasks/memory usage)
>> 
>> These counters cannot be totaled/merged.
> 
> So, these are handled outside of the statistics daemon, right?

separate couters and total couters may be both usable.

>> |   * file based statistics (but not needed if going to just XML)
>> |   * per zone statistics
>> 
>> number of zones may be large number.
>> need to define how to add/remove zones to collect zone's statistics.
> 
> Not disagreeing... we also need to decide on the default. I think we
> should turn zone statistics "on" by default, since I think most servers
> won't have a huge number of zones and may be interested.

AuthSrv module may have query counters for all zones, but I don't think
the minimal set includes all zone counters.

>> | == Ideas ==
>> | 
>> |   * per remote host statistics
>> 
>> need to consider how to collect.
>> 
>> # I use querylog to evaluate per remote host statistics.
>> 
>> |   * RESTful interface to get to specific statistics in XML (not everything at once)
>> 
>> need to define output interface
> 
> Well... output is mostly done... I think we use the BIND 9.7 interface.

Ok, it should be compatible with BIND 9.7.
Then, minimal couter set should include all BIND 9.7 couters.

>> |   * all BIND 10 components may have statistics counters (even if not DNS related)
>> 
>> need to define.
> 
> Yes, and probably out of scope for the statistics daemon itself.
> 
>> |   * Separate daemon to handle HTTP requests and/or generate XML reports; BIND 10 will have separate processes running so having a central location to submit counter information might be useful.
>> 
>> Is it statistics daemon?
> 
> Based on your description, I think this is a separate program.
> 
>> |   * will the daemon store details on all statistics?
>> 
>> Internal database ? external database ?
> 
> I think initially via an internal, SQLite, database. At some point
> someone will want these pushed into a "real" database and we'll want to
> make a way of connecting to arbitrary SQL back-ends, but I think it is
> best to implement that when the need arises. (Or even better, have a
> patch submitted when this happens.)

I agree.

-- 
Kazunori Fujiwara, JPRS