[bind10-dev] a strawman proposal of generic statistics spec

[JINMEI Tatuya / 神明達哉]

At Tue, 30 Oct 2012 17:33:09 -0700,
JINMEI Tatuya <jinmei at isc.org> wrote:

> This will require some intensive design discussions.  I'm not yet
> confident about how we should address this, but for a starter I'll
> send a strawman proposal in a followup message.

So, this is my initial thoughts.

After thinking about this for a while, I've been getting the
impression that one major architectural issue is that we are mixing
the specification and the data.  When we consider per RR type
statistics counters like this:

server/incoming_queries/A
server/incoming_queries/NS
...
server/incoming_queries/AAAA
and so on

conceptually the specification is just
server/incoming_queries/<per-RR-type-counter>, and A, NS, AAAA, etc
are kind of data.  The currently proposed implementation includes
things like A, AAAA, etc as part of the spec, which I suspect causes
many issues.

Also, at least in the current way of using module specs, it's not easy
to share the same concept in multiple modules.  This leads to another
concern that we'll have to repeat the same thing for the resolver.  We
may be able to do a similar thing as we share the data source
configuration from multiple modules, but I'm not so sure if that
solves all issues (such as repeating the same thing in the unified
spec).

So my basic idea is to separate the "data" part from the statistics
specification.  In the module spec, we only define a path to specific
counters, and define the actual counters outside of the spec, maybe as
part of an independent library.

A specific spec in the case of b10-auth would look like this:
 "statistics": [
	 { "item_name": "server",
	   "item_type": "any" // this is statistics-data
	 },
	 { "item_name": "classes",
	   "item_type": "list",
	   "list_item_spec":
	   {  "item_name": "class-statistics",
	      "item_type": "map",
	      "map_item_spec": [
		      { "item_name": "class",
			"item_type": "string" },
		      { "item_name": "zones",
			"item_type": "list",
			"list_item_spec":
			{ "item_name": "zone-statistics",
			  "item_type": "map",
			  "map_item_spec": [
				  { "item_name": "origin",
				    "item_type": "string" },
				  { "item_name": "statistics",
				    "item_type": "any" // statistics-data
				  }
			  ]
			}
		      }
	      ]
	   }
	 }
 ]

The "any" part defines (a set of) actual counters, whose spec is
outside of the spec file.  I'd envision it's a list of "statistics
specs", each of which would be a map like this:

{ "name": "requests",
  "type": "module",
  "module_name": "dns_server.request_stat",
  "data": <type-dependent-data>
}

The "module_name" attribute specifies (some kind of) the library that
understands the "request" statistics counters.  The authoritative
server would maintain an instance of something like
"request_stat::Counters" class, store intermediate statistics counters
there:

   request_stat::Counters request_counters;
   ...
   request_counters.increment(REQUEST_IPV6);
   request_counters.increment(REQUEST_EDNS0);
   ...etc,

and creates a corresponding JSON form to send it to the statics
daemon:
   request_counters.toJSON();

On the other hand, the statistics daemon first retrieves the spec,
and figures out the "any" type item should be a list of "statistics
specs", and load corresponding Python module from it:

   stat_spec_mod = imp.load(auth_spec[server][0]['module_name'])
   counters = stat_spec_mod.create_counters()

and when it gets counters from the authoritative server, it simply
passes the given data to the corresponding counters object:

   counters.update(auth_stat_data['server'][0][data'])

and/or retrieve it in the form of JSON, XML, etc.

We should be able to write a unified C++ library for the counter
module/class, and provide a thin Python wrapper for Python programs.

---
JINMEI, Tatuya