approach on parsing the query-log file
jpetersson at garnser.se
Wed Apr 29 15:35:39 UTC 2009
Thanks for the tip, however the main problem that I'm seeing is that
perl + MySQL becomes a bottle-neck if this approach were to be used. I
ran some tests yesterday showing that caching 500k rows in a variable
and send it to MySQL was 10 times as effective (90k vs 9k) than doing
I guess I could create an internal buffer in the script caching the
last X amount of messages based on a dynamic variable that adopts to
the query-flow and then creates a fork that writes it to the DB.
On Wed, Apr 29, 2009 at 12:44 AM, Chris Dew <cmsdew at googlemail.com> wrote:
> You may be interested in using circular buffers, instead of a log file.
> I've used emlog successfully in the past and been very pleased with
> it's performance.
> Hope this is useful.
> 2009/4/29 Scott Haneda <talklists at newgeo.com>:
>> I have read the other posts here, and it looks like you are setting on tail,
>> or a pipe, but that log rotation is causing you headaches.
>> I have had to deal with things like this in the past, and took a different
>> approach. Here are some ideas to think about.
>> Since you mentioned below you wanted this in real time, and that parsing an
>> old log file is out, what about setting up a second log in named, of the
>> same data, but do not rotate the log at all?
>> This gives you a log that you can run tail on. It probably is going to grow
>> too large. I solved this for a different server in the past, by telling the
>> log that was a clone to be be limited in size. In this way, it was not
>> rolled out, but rather, truncated.
>> I am not sure how named would do this. If it will not truncate it, you can
>> write a small script to do it for you. Now that you have a log that is
>> maintained at a fixed size that is manageable, you can do your tail business
>> on it.
>> I also seem to remember, tail has some flags that may help you with dealing
>> with the log ration issues. I only remember them vaguely, as they were not
>> applicable to what I was doing at the time.
>> Hope this helps some.
>> On Apr 27, 2009, at 10:26 PM, Jonathan Petersson wrote:
>>> Hi all,
>>> I'm thinking of writing a quick tool to archive the query-log in a
>>> database to allow for easier reports.
>>> The obvious question that occurs is; What would be what's the best
>>> approach to do this?
>>> Running scripts that parses through the query-log would cause locking
>>> essentially killing BIND on a heavy loaded server and only parsing
>>> archived files wouldn't allow real-time information, also re-parsing
>>> the same set of data over and over again until the log has rotated
>>> would cause unnecessary I/O load. I'm guessing the best would be to
>>> have BIND write directly to a script that dumps the data where-ever it
>>> makes sense to.
>>> I've used BIND statistics and found it highly useful but then again it
>>> doesn't allow me to make breakdowns based on host/query.
>>> If anyone has done something like this or having pointers on how this
>>> could achieved any information is welcome!
>> Scott * If you contact me off list replace talklists@ with scott@ *
>> bind-users mailing list
>> bind-users at lists.isc.org
> bind-users mailing list
> bind-users at lists.isc.org
More information about the bind-users