[bind10-dev] Tracking queries and other data, was: NSAS Using Authority/Additional Information?

[Shane Kerr]

All,

On Thu, 2010-12-02 at 10:55 +0100, Michal 'vorner' Vaner wrote:
> Do you think that requirement for a cache is sane? And that passing this around
> is OK? In the todays call, Shane mentioned tagging entries, so we will need to
> carry some kind of state object with us anyway, so maybe being able to put the
> cookie into that (so that NSAS code could be completely oblivious about the
> cookie existence).

Without answering the particular question, let me clarify what I am
talking about.

Resolution is a complicated and relatively slow process (sometimes
taking seconds). A single client query may result in tens of individual
packets being sent by the resolver on behalf of that client (an example
using one of my zones is at the end).

Things are complicated even further because some of the entries may be
in cache.

Also, different clients may ask the same question at the same time, and
a smart server will not try to re-ask the same questions to get the
answer. Plus clients may ask *related* questions (for example someone
wants MX for ISC.ORG and someone else wants AAAA for WWW.ISC.ORG, which
can be joined until ISC.ORG's name servers are found). A smart server
will consolidate as many queries as possible.


In the context of logging, if you want to make sense out of what is
going on I think you need to have some unique identifier that you can
include in each log message that reflects the various events that the
log message may apply to. Please ignore the specific syntax, but
something like this:

        client query WWW.ISC.ORG AAAA                <- query arrives
          new query tag: q20101202T163557.034731-00
        WWW.ISC.ORG AAAA not in cache 
          query tag: q20101202T163557.034731-00
        ISC.ORG NS not in NSAS 
          query tag: q20101202T163557.034731-00
        client query ISC.ORG MX                      <- another query
        arrives
          new query tag: q20101202T163557.034740-00
        ORG NS in NSAS
          query tag: q20101202T163557.034731-00
          query tag: q20101202T163557.034740-00
        query 2001:500:f::1 for WWW.ISC.ORG AAAA
          query tag: q20101202T163557.034731-00
          query tag: q20101202T163557.034740-00
        answer received for WWW.ISC.ORG AAAA, ISC.ORG added to NSAS
          query tag: q20101202T163557.034731-00
          query tag: q20101202T163557.034740-00
        query AMS.SNS-PB.ISC.ORG at 2001:500:60::30 for WWW.ISC.ORG AAAA
          query tag: q20101202T163557.034731-00
        query AMS.SNS-PB.ISC.ORG at 199.6.1.30 for ISC.ORG MX
          query tag: q20101202T163557.034740-00
        answer received for ISC.ORG MX
          query tag: q20101202T163557.034740-00
        answer received for WWW.ISC.ORG AAAA
          query tag: q20101202T163557.034731-00

The idea here is that you can see each action done and the queries that
it is done for. We may actually want to expand this a bit and include
various kinds of tags...

        client query WWW.ISC.ORG A
          new query tag: q20101202T163557.035117-00
        query 199.NS.ISC.AFILIAS-NST.INFO at 254.63.254 for WWW.ISC.ORG A
          query tag: q20101202T163557.035117-00
        answer received for WWW.ISC.ORG A
          new answer tag: a20101202T163557.038471-00
          query tag: q20101202T163557.035117-00
        reply sent to user for WWW.ISC.ORG A
          query tag: q20101202T163557.035117-00
        cache updated for WWW.ISC.ORG A
          answer tag: a20101202T163557.038471-00

Anyway, my thinking is that a design for a logging system should allow
for such tracking. This will help administrators answer questions like
"why is my system sending packets to machine X?" or "why am I never
seeing any packets to my authoritative server Y?"



Simple example of resolution follows (skip to the last two paragraphs at
the end if you don't want to double-check my work):

Consider someone who wants to send a mail to my test account,
TIME-TRAVELLERS.NL.EU.ORG. I think this means that starting from an
empty cache means:

1. query to root -> yields NS of ORG, plus glue

2. query to one of ORG servers -> yields NS of EU.ORG plus *some* glue

Note that the EU.ORG servers include servers in .NET, .ORG, and .FR.
Only the two ORG servers will have glue returned, and only 1 has an IPv4
address. Lets be bold and assume that NS0.PASTEUR.FR was chosen instead,
which means:

3. query to root -> yields NS of FR, plus glue

4. query to one of FR servers -> yields NS of PASTEUR.FR plus glue

Now the glue here also happens to include NS0.PASTEUR.FR, but we need to
check the child anyway.

5. query to one of PASTEUR.FR servers -> yields no A for NS0.PASTEUR.FR

Crap, no IPv6 support. Maybe IPv4?

6. query to one of PASTEUR.FR servers -> yields A for NS0.PASTEUR.FR

Okay, now we can get back to trying to find that MX record!

7. query to NS0.PASTEUR.FR -> yields NS for NL.EU.ORG plus glue

It turns out that NS0.PASTEUR.FR is actually an open resolver, but if
you set +norecurse in dig you get this. :)

To be honest, I'm not sure what happens here. I guess since all of the
NS except NS.EU.ORG are out of bailiwick we ignore the glue and start
resolution from the top for any of the NS here.

Lets assume we pick NS1.NICON.NL as our name server. This means

8. query to root (again!) -> yields NS for NL plus glue

9. query to one of NL servers -> yields NS for NICON.NL plus *some* glue

Lets assume we pick one of the NL servers with glue...

10. query to one of NICON.NL servers -> yields an AAAA record

I don't know if we try to find both AAAA and A records here... I guess
so?

11. query to one of NICON.NL servers -> yields an A record

Now, back to trying to get that MX record!

12. query to NS1.NICON.NL -> yields NS for TIME-TRAVELLERS.NL.EU.ORG 

Yay! No glue though. Two are in .NET and one is in .ORG... probably
we'll end up with .NET, so finding MADRAS.CURRYBOYS.NET means going to
the root again...

13. query to root -> yields NS for NET plus glue

14. query to one of NET servers -> yields CURRYBOYS.NET servers plus
*some* glue (since they also have a mix of NET and ORG, with SNS acting
as a secondary)

The glue is actually MADRAS.CURRYBOYS.NET, but again we need to query
below the zone cut (right?), so assuming that we get lucky and go to
MADRAS.CURRYBOYS.NET to find it's own address...

15. query to MADRAS.CURRYBOYS.NET -> no AAAA for MADRAS.CURRYBOYS.NET

And since we need an address, we'll try an A record...

16. query to MADRAS.CURRYBOYS.NET -> A for MADRAS.CURRYBOYS.NET

Yay! Now back to that MX lookup...

17. query to MADRAS.CURRYBOYS.NET -> MX for TIME-TRAVELLERS.NL.EU.ORG

Wow. That was easy. Of course, at this point we only have a hostname for
the MX, so we'll probably want to look that up, but that would be a
separate client query.

So, 17 queries to authoritative servers to look up one MX record. It
could have been more, it could have been less, depending on which
servers are chosen. And this is without any drops, lameness, firewall
nonsense, and so on. AND without anything in cache.

In real life it is actually way worse than this... BIND 9 does an AAAA
and an A lookup for every host in each NS RRSET it encounters. I did a
tcpdump on an idle host and discovered that over 90 queries were made
for this single MX lookup on a fresh BIND 9 startup. This probably makes
sense given that it should reduce latency for users and DNS traffic is
relatively cheap, but it does complicate the process of figuring out
what is going on by looking at simple packet traces. Trying to figure
out what authoritative queries are for which recursive client queries is
almost impossible without some help from inside the recursor!

--
Shane