Bind 9.11 serving up false answers for a single domain. (OT)

Wed Feb 10 23:23:56 UTC 2021

<slightly-pointless-comment-in-defence-of-us-zone>

If you look closer, you’ll see that ‘us.’ is RSASHA256. ‘state.ma.us.’ however, is delegated to the state officials of the Commonwealth of Massachusetts and is indeed RSASHA1NSEC3.

Stuart
... one of the guy’s that does the DNSSEC for US TLD.

From: bind-users <bind-users-bounces at lists.isc.org> on behalf of "John W. Blue via bind-users" <bind-users at lists.isc.org>
Reply to: "John W. Blue" <john.blue at rrcic.com>
Date: Thursday, 11 February 2021 at 9:21 am
To: bind-users <bind-users at lists.isc.org>
Subject: RE: Bind 9.11 serving up false answers for a single domain.

Notice: This email is from an external sender. 

Three words:  tcpdump and wireshark

It is like peanut and jelly .. hall and oates .. salt and pepper .. ebb and flow .. pen and paper .. I could go on but … 

Know them.  Love them.  They are your newest best friends.

<grin>

Using tcpdump IMHO should be the first tool anyone uses when troubleshooting seemly unexplainable DNS weirdness.

Knowing what is being put on the wire (or lack thereof) is critical since it provides key factual data points that decisions can be made on.  When running tcpdump on the DNS server I personally prefer this command:

tcpdump -n -i <interface eg eth0> -s 65535 -w <filename.pcap>

dash n is telling tcpdump that you do not want it to resolve hostnames.  This is an important option when doing DNS troubleshooting because you do not want extra resolutions taking place.
dash s is saying gimme the full packet.
dash w is the name of the file you want the output saved in.

After starting the command, run several queries from a host and ctrl+c to exit.

Once you get your file into wireshark now you can start slicing n dicing on the data!

Here is handy wireshark filter:  dns.qry.name == internet-dns1.state.ma.us

By using a filter of dns.flags.rcode == (number here) you can drive off into the weeds and get super granular with sorting the data.  For example “dns.flags.rcode == 2” will show you all of the server failures for queries.

It is hard to provide further guidance on what to do since what you find in the pcap is only a starting point.

Good hunting!

As an aside I would like to mention that you do not need to travel home to get situational awareness when the diggui.com website can be used instead.

Also.  For the people running .us tld .. SHA1 for DNSSEC .. really?

https://dnsviz.net/d/state.ma.us/dnssec/

John

From: bind-users [mailto:bind-users-bounces at lists.isc.org] On Behalf Of sami's strat
Sent: Wednesday, February 10, 2021 11:54 AM
To: Mark Andrews
Cc: bind-users
Subject: Re: Bind 9.11 serving up false answers for a single domain.

Thank you all for responding.  One final query about this. I'm seeing this issue on my production servers at work.  Yet, when I run the same queries at home, I don't see those failed queries.  I actually flushed DNS cache, cleared Linux O/S cache, and even bounced my personal DNS server trying to reproduce the issue.  But I could not.

TIA

On Wed, Feb 10, 2021 at 12:09 AM Mark Andrews <mailto:marka at isc.org> wrote:
Run ‘dig +trace +all http://internet-dns1.state.ma.us’ which will show you the glue
records then try ‘dig +dnssec +norec http://internet-dns1.state.ma.us @<address>’ for
all the addresses in the glue records.

e.g.
        dig +dnssec +norec http://internet-dns1.state.ma.us @http://146.243.122.17

Mark

> On 10 Feb 2021, at 14:50, sami's strat <mailto:sami.strat at gmail.com> wrote:
> 
> Thanks Mark.
> 
> However, the traceroute to the hostnamed failed for the same reason.  Please note:
> 
> [root at myhost data]# dig http://internet-dns1.state.ma.us
>  
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> http://internet-dns1.state.ma.us
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 61641
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
>  
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;http://internet-dns1.state.ma.us.     IN      A
>  
> ;; Query time: 1263 msec
> ;; SERVER: 192.168.33.12#53(192.168.33.12)
> ;; WHEN: Tue Feb 09 22:34:15 EST 2021
> ;; MSG SIZE  rcvd: 54
>  
> [root at myhost data]# dig http://internet-dns1.state.ma.us +trace
>  
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> http://internet-dns1.state.ma.us +trace
> ;; global options: +cmd
> .                       516485  IN      NS      http://c.root-servers.net.
> .                       516485  IN      NS      http://e.root-servers.net.
> .                       516485  IN      NS      http://f.root-servers.net.
> .                       516485  IN      NS      http://l.root-servers.net.
> .                       516485  IN      NS      http://m.root-servers.net.
> .                       516485  IN      NS      http://d.root-servers.net.
> .                       516485  IN      NS      http://g.root-servers.net.
> .                       516485  IN      NS      http://k.root-servers.net.
> .                       516485  IN      NS      http://b.root-servers.net.
> .                       516485  IN      NS      http://h.root-servers.net.
> .                       516485  IN      NS      http://a.root-servers.net.
> .                       516485  IN      NS      http://i.root-servers.net.
> .                       516485  IN      NS      http://j.root-servers.net.
> .                       516485  IN      RRSIG   NS 8 0 518400 20210222230000 20210209220000 42351 . QCzDH8eHlHVbx4SxIIwk8xnk6ky/q+zRh8KAUfI98lqHcIP4NLxzCe6f mC2sNX1VcthEy6Lwnobm8OyJCRpNEHedYrS01aMhAVzUfM+/PJ9MWn0w SkmXxyZMJZXF/kl4GDNX0x/GW3+DkeTeZI9+B540Yvj47qJv2bD9nIQG NtE7bDze7bgMJkIuBlEzPfwp7YW5ud8qdC6HdUoEMqygwZcWAiQu8gpb q21z8W5hcdci1OouDFytNWrXAvfSsuR635+GzSj+RZjYo+447uP7lKsK N5aeVQ/BPh5jM32xVO+zwyp7v9Nky1vSP/BchMQ/3cqg3Ee7zobl8OQd CSd/SA==
> ;; Received 1097 bytes from 192.168.33.12#53(192.168.33.12) in 0 ms
>  
> us.                     172800  IN      NS      http://a.cctld.us.
> us.                     172800  IN      NS      http://b.cctld.us.
> us.                     172800  IN      NS      http://c.cctld.us.
> us.                     172800  IN      NS      http://e.cctld.us.
> us.                     172800  IN      NS      http://f.cctld.us.
> us.                     172800  IN      NS      http://k.cctld.us.
> us.                     86400   IN      DS      21364 8 1 260D0461242BCF8F05473A08B05ED01E6FA59B9C
> us.                     86400   IN      DS      21364 8 2 B499CFA7B54D25FDE1E6FE93076FB013DAA664DA1F26585324740A1E 6EBDAB26
> us.                     86400   IN      RRSIG   DS 8 1 86400 20210222230000 20210209220000 42351 . rujvGB0s2bsqzBuzRliH6QK9vH84ETZV7gZMEhJyzMFofWhj9ZZaNWE/ VvdA9rC16IOEocvARv2rOqk7G3KTzdkHHZcwcZSQyVqsOIaIywGFuEgd viSXF6+M5MocUgEMp5dtt6SBLHG+lE/FV/3HylKSHsxdO/F6PeWKgcBZ D4lZQ6w5asmlbdKJKMhlWPp6UaxBE7ACaxndBQixoNqXQuPrXpXi1Fnj ntFtTfn57hMyrdTojIJ8X7/HKjCrbm3CL/WJ+VZR051OGCdZVjpUaDXR x7G9lDhu3K5clar9PGYyUJM7+RBKzrQJep7HrjL2nZdoTyfY4i33S+EZ sTlTOA==
> ;; Received 707 bytes from 199.7.91.13#53(http://d.root-servers.net) in 4 ms
>  
> http://state.ma.us.            7200    IN      NS      http://internet-dns3.state.ma.us.
> http://state.ma.us.            7200    IN      NS      http://internet-dns1.state.ma.us.
> http://state.ma.us.            7200    IN      NS      http://internet-dns2.state.ma.us.
> http://state.ma.us.            3600    IN      DS      47628 7 2 5379F9F747214E5A63416775396BCFF98FA4867AE66E09BCBEBE0DCC 1682C369
> http://state.ma.us.            3600    IN      DS      41388 7 1 36D899932AF794EADD671161515E48FE829BB7FE
> http://state.ma.us.            3600    IN      DS      41388 7 2 BBAB433D3853571F42516E70659AF1F85FA4FBA0FDFCEAD4D092592A 00C78769
> http://state.ma.us.            3600    IN      DS      47628 7 1 485E0EE2F7C08FCE51D1E284321242930274833A
> http://state.ma.us.            3600    IN      RRSIG   DS 8 3 3600 20210307200856 20210205191212 53985 us. O8KqBHzlZsDqrZi0NQO4JEiN0b8j04/Lb8W2uVz5PyrAat1VgZKQ3Ws6 6PNtbZDMv6YX6QA8fWFLxNmeJ1/4L3wLu8EKYXaThA9Zxll7mKFj1iPf nqiVq5hOo8Ul3inmfM/tjCQ21IHc/v0JZygZNd/h0SxXWlQXi+W3G9LN +4z/qxtl9dGD1ka54Ln3MAVxB1Tp4pt0ri4qPLmfGKf/HA==
> couldn't get address for 'http://internet-dns3.state.ma.us': not found
> couldn't get address for 'http://internet-dns1.state.ma.us': not found
> couldn't get address for 'http://internet-dns2.state.ma.us': not found
> dig: couldn't get address for 'http://internet-dns3.state.ma.us': no more
> [root at myhost data]#
> 
> On Tue, Feb 9, 2021 at 10:10 PM Mark Andrews <mailto:marka at isc.org> wrote:
> Well you could try tracing the addresses of the nameservers for which
> there where errors reported.  It could be as simple as a routing issue
> between you and these servers.
> 
> > On 10 Feb 2021, at 13:25, sami's strat <mailto:sami.strat at gmail.com> wrote:
> > 
> > couldn't get address for 'http://internet-dns1.state.ma.us': not found
> > couldn't get address for 'http://internet-dns3.state.ma.us': not found
> > couldn't get address for 'http://internet-dns2.state.ma.us': not found
> > dig: couldn't get address for 'http://internet-dns1.state.ma.us': no more
> 
> Yet, I do this on my personal computer at home, and it works without an issue.
> 
> Any other thoughts?  TIA 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: mailto:marka at isc.org