Bind Queries log file format

[Larry Stone]

I’ve been around long enough to remember when upward compatability was something that was expected. A program built to work with the current version of data (e.g. data records, log records, whatever) or even a shared library was expected to be able to continue to work with all future versions without the need for changes or rebuilding/recompiling. For data, that meant new fields went on the end of the record so that anything expecting the old format still found everything where it always was and the new stuff was at the end of the record where the old programs never even looked. But sadly, this appears to be a lost art these days.

Where I work, we have a data set that has 20 years of data in it. Over the years, the record length was extended but once a field was placed at a given point in the record, it never, ever moved so that programs written years ago that had no need for the new fields still ran just fine. And with hundreds if not thousands of programs around the company that read this data set, insuring that format changes did not break things was a very high priority. Occasionally, fields went away in which case that spot in the record was just left blank for all new records.

For the BIND log records described below, what I describe appears to be what was done through 9.10.0. But then at 9.11.0, we have a field in the middle of the record being changed (EDNS changed to EDNS+version). What, IMHO, should have been done here was to put the version on the end (either going with a split filed - EDNS in one place and the version in another or by duplicating EDNS by having it without version where it was and then again with version on the end of the record) so that old programs parsing the log file still worked. So instead of:
> 9.10.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.11.0: client, qname, qclass, qtype, RD, signed, EDNS + version, TCP, DO, CD, cookies, local address
> 9.12.0: client, qname, qclass, qtype, RD, signed, EDNS + version, TCP, DO, CD, cookies, local address, ecs


it should have been
> 9.10.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.11.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, cookies, local address, EDNSversion
> 9.12.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, cookies, local address, EDNSversion, ecs

-- 
Larry Stone
lstone19 at stonejongleux.com





> On Feb 7, 2017, at 9:06 PM, Mark Andrews <marka at isc.org> wrote:
> 
> 
> In message <DB6PR0501MB2309198D18749297D7EBAE41C0420 at DB6PR0501MB2309.eurprd05.p
> rod.outlook.com>, Paul Roberts writes:
>> I have to say I agree with the approach of putting this extra info into a sep
>> arate file. I appreciate this could cause additional problems (disk utilisati
>> on, extra I/O's, log rolling etc.) but I would prefer to keep the query log f
>> ormat as stable as possible. I am still mopping up the last big change when I
>> SC added the FQDN reference at the start of each message and I'm getting a li
>> ttle tired of dealing with customers and their broken regex's when log format
>> s change because they've upgraded BIND.
>> 
>> There are also wider implications - there are products out there that hard co
>> de the regex and it can't be modified, so that then requires dealing with ven
>> dors, submitting bug reports/enhancement requests, providing evidence, busine
>> ss impact statements, also I have to perform root cause analysis for customer
>> s why their SIEM is no longer capturing the logs, which can have serious regu
>> latory implications and consequences (banks etc.), then there's testing every
>> upgrade in the lab before we run in production etc., I have enough work on m
>> y plate as it is! :-)
>> 
>> Basically there's a whole world of pain out there that can be avoided if you 
>> just keep the log format the same. :-)
>> 
>> Thanks,
>> 
>> Paul
> 
> Change happens.  The DNS protocol has expanded enormously from the
> original specification.  To expect the summary log line to not
> change with that change is just not realistic.  We do try to keep
> the format change to a .0 release.  This one we missed.
> 
> We currently log:
> 
> client, qname, qclass, qtype, RD (+/-), was the request signed (S),
> the EDNS with version, was it over TCP (T), was DO=1 set (D), was
> CD=1 set (C), were DNS COOKIES in use and was it a valide server
> cookie or just a client cookie (V, K).  We log the interface it was
> received on and if the ECS option.
> 
> Not everyone wants all of these details but someone wants everyone
> of these.
> 
> 9.1.0: client, qname, qclass, qtype
> 9.2.0: client, qname, qclass, qtype
> 9.3.0: client, qname, qclass, qtype, RD, signed, EDNS
> 9.4.0: client, qname, qclass, qtype, RD, signed, EDNS
> 9.5.0: client, qname, qclass, qtype, RD, signed, EDNS, DO, CD
> 9.6.0: client, qname, qclass, qtype, RD, signed, EDNS, DO, CD
> 9.7.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.8.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.9.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.10.0: client, qname, qclass, qtype, RD, signed, EDNS, TCP, DO, CD, local address
> 9.11.0: client, qname, qclass, qtype, RD, signed, EDNS + version, TCP, DO, CD, cookies, local address
> 9.12.0: client, qname, qclass, qtype, RD, signed, EDNS + version, TCP, DO, CD, cookies, local address, ecs
> 
> That's basically 5 changes in 17 years.
> 
> Mark
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
> 
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users