[bind10-dev] Rate limiting for logging?

[Mukund Sivaraman]

On Mon, Jul 15, 2013 at 09:04:30AM +0200, Michal 'vorner' Vaner wrote:
> If we wanted to solve this problem on the log4cplus level, we would need to
> somehow detect the same messages, which will be hard, because they don't have to
> be exactly the same (with different parameters, for example slightly different
> domain name or source port of query). So we need some kind of detection of

We need to first figure out what are candidates for suppressing. Is it
100% identical messages? Should we suppress even messages with different
argument values? What behavior does the sysadmin want? This will point
us to what approach we need to use.

----

For suppressing identical messages with identical argument values, we
could have something like the following:

* Maintain a hashtable where the key is the message text and it maps to
  an entry containing (apart from the key) a timestamp and count.

* At every log attempt (part B):

  + If message is already logged, we increment its count by 1 and don't
    output it.

  + If message is not found in the hashtable, we initialize an entry for
    it with {time=now, count=1} and output it.

* At every log attempt (part A): if 1 second has passed since last
  sweep, we sweep the hashtable. For each entry that is older than 1
  second in the hashtable and its count > 1, we output that that message
  was repeated count-1 times in the last second, and remove it from the
  hashtable. This way we can also print the number of times duplicates
  have occured. The upper limit on the size of the hashtable is < the
  number of distinct messages that are logged in 2 seconds.

  The sweep can be optimized by using a linked-list (new entries in time
  are appended, and oldest entries are at the head), and the hashtable
  as a secondary index to update the counter.

		Mukund
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20130716/7015d7ca/attachment.bin>