approach on parsing the query-log file

[Jonathan Petersson]

So I gave tail a try in perl both via File::Tail and by putting tail
-f in a pipe. Neither seems to be handling the logrotation well. In my
case I'm running a test sending 1 million queries, of those half is
picked up by File::Tail if you define how often it should re-read the
file but using tail -f straight or File::Tail without arguments just
stops once the log has rotated as it doesn't seam to figure out to
continue onto the new file.

/Jonathan

On Tue, Apr 28, 2009 at 8:52 AM, David Forrest <drf at maplepark.com> wrote:
> On Tue, 28 Apr 2009, Gregory Hicks wrote:
>
>>
>>> From: Jonathan Petersson <jpetersson at garnser.se>
>>> Date: Tue, 28 Apr 2009 08:13:25 -0700
>>> Subject: Re: approach on parsing the query-log file
>>> To: Niall.oReilly at ucd.ie
>>> Cc: Bind Mailing <bind-users at lists.isc.org>
>>>
>>> Yeah I've thought about using tail but I'm not sure how locking would
>>> be managed when logrotate kicks in, does anyone know?
>>
>> I use "tail -f <log-file>"
>>
>> When the log rotates, the tail is still running against the rotated
>> file.  I have to manually change to the current file. ("^C-!!" works)
>>
>> A better way to do it might be to have the 'logfile' be a pipe and have
>> the parsing intelligence on the other side of the pipe.  Have the log
>> rotation "smarts" be on the other side of the pipe also.  (At one $JOB,
>> I used this technique to separate out different log messages from
>> simultaneously running SMTP processes.)
>>
>> Regards,
>> GRegory Hicks
>>>
>>> On Tue, Apr 28, 2009 at 3:41 AM, Niall O'Reilly <Niall.oReilly at ucd.ie>
>>
>> wrote:
>>>>
>>>> On Mon, 2009-04-27 at 22:26 -0700, Jonathan Petersson wrote:
>>>>>
>>>>> The obvious question that occurs is; What would be what's the best
>>>>> approach to do this?
>>>>
>>>>        I've not used it, but a colleague is very keen on File::Tail
>>>>        (http://search.cpan.org/~mgrabnar/File-Tail-0.99.3/Tail.pm).
>>>>        Apparently, it looks after log-file roll-over and 'just
>>
>> works'.
>>>>
>>>>        /Niall
>>>>
>>>>
>>>>
>
> I use tail --follow=name <logfile> as the tail then switches to the new
> inode.  An alternative is to to put the copytruncate directive in
> /etc/logrotate.conf as the possible loss of one or two queries is usually
> not significant to statistical analysis.  Using inotail (which is supposedly
> less processor intensive) requires the second approach as it does not
> include the --follow=name option.
>
> Dave
> --
> David Forrest                   e-mail   drf @ maplepark.com
> Maple Park Development Corporation  http://xen.maplepark.com
> St. Louis, Missouri