<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Three words: tcpdump and wireshark<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It is like peanut and jelly .. hall and oates .. salt and pepper .. ebb and flow .. pen and paper .. I could go on but …
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Know them. Love them. They are your newest best friends.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><grin><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Using tcpdump IMHO should be the first tool anyone uses when troubleshooting seemly unexplainable DNS weirdness.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Knowing what is being put on the wire (or lack thereof) is critical since it provides key factual data points that decisions can be made on. When running tcpdump
on the DNS server I personally prefer this command:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">tcpdump -n -i <interface eg eth0> -s 65535 -w <filename.pcap><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">dash n is telling tcpdump that you do not want it to resolve hostnames. This is an important option when doing DNS troubleshooting because you do not want
extra resolutions taking place.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">dash s is saying gimme the full packet.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">dash w is the name of the file you want the output saved in.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">After starting the command, run several queries from a host and ctrl+c to exit.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Once you get your file into wireshark now you can start slicing n dicing on the data!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Here is handy wireshark filter: dns.qry.name == internet-dns1.state.ma.us<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">By using a filter of dns.flags.rcode == (number here) you can drive off into the weeds and get super granular with sorting the data. For example “dns.flags.rcode
== 2” will show you all of the server failures for queries.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It is hard to provide further guidance on what to do since what you find in the pcap is only a starting point.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Good hunting!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As an aside I would like to mention that you do not need to travel home to get situational awareness when the diggui.com website can be used instead.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Also. For the people running .us tld .. SHA1 for DNSSEC .. really?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">https://dnsviz.net/d/state.ma.us/dnssec/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">John<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> bind-users [mailto:bind-users-bounces@lists.isc.org]
<b>On Behalf Of </b>sami's strat<br>
<b>Sent:</b> Wednesday, February 10, 2021 11:54 AM<br>
<b>To:</b> Mark Andrews<br>
<b>Cc:</b> bind-users<br>
<b>Subject:</b> Re: Bind 9.11 serving up false answers for a single domain.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Thank you all for responding. One final query about this. I'm seeing this issue on my production servers at work. Yet, when I run the same queries at home, I don't see those failed queries. I actually flushed DNS cache, cleared Linux
O/S cache, and even bounced my personal DNS server trying to reproduce the issue. But I could not.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">TIA<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Feb 10, 2021 at 12:09 AM Mark Andrews <<a href="mailto:marka@isc.org">marka@isc.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt">Run ‘dig +trace +all <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a>’ which will show you the glue<br>
records then try ‘dig +dnssec +norec <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a> @<address>’ for<br>
all the addresses in the glue records.<br>
<br>
e.g.<br>
dig +dnssec +norec <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a> @<a href="http://146.243.122.17" target="_blank">146.243.122.17</a><br>
<br>
Mark<br>
<br>
> On 10 Feb 2021, at 14:50, sami's strat <<a href="mailto:sami.strat@gmail.com" target="_blank">sami.strat@gmail.com</a>> wrote:<br>
> <br>
> Thanks Mark.<br>
> <br>
> However, the traceroute to the hostnamed failed for the same reason. Please note:<br>
> <br>
> [root@myhost data]# dig <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a><br>
> <br>
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a><br>
> ;; global options: +cmd<br>
> ;; Got answer:<br>
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 61641<br>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1<br>
> <br>
> ;; OPT PSEUDOSECTION:<br>
> ; EDNS: version: 0, flags:; udp: 4096<br>
> ;; QUESTION SECTION:<br>
> ;<a href="http://internet-dns1.state.ma.us" target="_blank">internet-dns1.state.ma.us</a>. IN A<br>
> <br>
> ;; Query time: 1263 msec<br>
> ;; SERVER: 192.168.33.12#53(192.168.33.12)<br>
> ;; WHEN: Tue Feb 09 22:34:15 EST 2021<br>
> ;; MSG SIZE rcvd: 54<br>
> <br>
> [root@myhost data]# dig <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a> +trace<br>
> <br>
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> <a href="http://internet-dns1.state.ma.us" target="_blank">
internet-dns1.state.ma.us</a> +trace<br>
> ;; global options: +cmd<br>
> . 516485 IN NS <a href="http://c.root-servers.net" target="_blank">
c.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://e.root-servers.net" target="_blank">
e.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://f.root-servers.net" target="_blank">
f.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://l.root-servers.net" target="_blank">
l.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://m.root-servers.net" target="_blank">
m.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://d.root-servers.net" target="_blank">
d.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://g.root-servers.net" target="_blank">
g.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://k.root-servers.net" target="_blank">
k.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://b.root-servers.net" target="_blank">
b.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://h.root-servers.net" target="_blank">
h.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://a.root-servers.net" target="_blank">
a.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://i.root-servers.net" target="_blank">
i.root-servers.net</a>.<br>
> . 516485 IN NS <a href="http://j.root-servers.net" target="_blank">
j.root-servers.net</a>.<br>
> . 516485 IN RRSIG NS 8 0 518400 20210222230000 20210209220000 42351 . QCzDH8eHlHVbx4SxIIwk8xnk6ky/q+zRh8KAUfI98lqHcIP4NLxzCe6f mC2sNX1VcthEy6Lwnobm8OyJCRpNEHedYrS01aMhAVzUfM+/PJ9MWn0w SkmXxyZMJZXF/kl4GDNX0x/GW3+DkeTeZI9+B540Yvj47qJv2bD9nIQG
NtE7bDze7bgMJkIuBlEzPfwp7YW5ud8qdC6HdUoEMqygwZcWAiQu8gpb q21z8W5hcdci1OouDFytNWrXAvfSsuR635+GzSj+RZjYo+447uP7lKsK N5aeVQ/BPh5jM32xVO+zwyp7v9Nky1vSP/BchMQ/3cqg3Ee7zobl8OQd CSd/SA==<br>
> ;; Received 1097 bytes from 192.168.33.12#53(192.168.33.12) in 0 ms<br>
> <br>
> us. 172800 IN NS <a href="http://a.cctld.us" target="_blank">
a.cctld.us</a>.<br>
> us. 172800 IN NS <a href="http://b.cctld.us" target="_blank">
b.cctld.us</a>.<br>
> us. 172800 IN NS <a href="http://c.cctld.us" target="_blank">
c.cctld.us</a>.<br>
> us. 172800 IN NS <a href="http://e.cctld.us" target="_blank">
e.cctld.us</a>.<br>
> us. 172800 IN NS <a href="http://f.cctld.us" target="_blank">
f.cctld.us</a>.<br>
> us. 172800 IN NS <a href="http://k.cctld.us" target="_blank">
k.cctld.us</a>.<br>
> us. 86400 IN DS 21364 8 1 260D0461242BCF8F05473A08B05ED01E6FA59B9C<br>
> us. 86400 IN DS 21364 8 2 B499CFA7B54D25FDE1E6FE93076FB013DAA664DA1F26585324740A1E 6EBDAB26<br>
> us. 86400 IN RRSIG DS 8 1 86400 20210222230000 20210209220000 42351 . rujvGB0s2bsqzBuzRliH6QK9vH84ETZV7gZMEhJyzMFofWhj9ZZaNWE/ VvdA9rC16IOEocvARv2rOqk7G3KTzdkHHZcwcZSQyVqsOIaIywGFuEgd viSXF6+M5MocUgEMp5dtt6SBLHG+lE/FV/3HylKSHsxdO/F6PeWKgcBZ
D4lZQ6w5asmlbdKJKMhlWPp6UaxBE7ACaxndBQixoNqXQuPrXpXi1Fnj ntFtTfn57hMyrdTojIJ8X7/HKjCrbm3CL/WJ+VZR051OGCdZVjpUaDXR x7G9lDhu3K5clar9PGYyUJM7+RBKzrQJep7HrjL2nZdoTyfY4i33S+EZ sTlTOA==<br>
> ;; Received 707 bytes from 199.7.91.13#53(<a href="http://d.root-servers.net" target="_blank">d.root-servers.net</a>) in 4 ms<br>
> <br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 7200 IN NS
<a href="http://internet-dns3.state.ma.us" target="_blank">internet-dns3.state.ma.us</a>.<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 7200 IN NS
<a href="http://internet-dns1.state.ma.us" target="_blank">internet-dns1.state.ma.us</a>.<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 7200 IN NS
<a href="http://internet-dns2.state.ma.us" target="_blank">internet-dns2.state.ma.us</a>.<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 3600 IN DS 47628 7 2 5379F9F747214E5A63416775396BCFF98FA4867AE66E09BCBEBE0DCC 1682C369<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 3600 IN DS 41388 7 1 36D899932AF794EADD671161515E48FE829BB7FE<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 3600 IN DS 41388 7 2 BBAB433D3853571F42516E70659AF1F85FA4FBA0FDFCEAD4D092592A 00C78769<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 3600 IN DS 47628 7 1 485E0EE2F7C08FCE51D1E284321242930274833A<br>
> <a href="http://state.ma.us" target="_blank">state.ma.us</a>. 3600 IN RRSIG DS 8 3 3600 20210307200856 20210205191212 53985 us. O8KqBHzlZsDqrZi0NQO4JEiN0b8j04/Lb8W2uVz5PyrAat1VgZKQ3Ws6 6PNtbZDMv6YX6QA8fWFLxNmeJ1/4L3wLu8EKYXaThA9Zxll7mKFj1iPf
nqiVq5hOo8Ul3inmfM/tjCQ21IHc/v0JZygZNd/h0SxXWlQXi+W3G9LN +4z/qxtl9dGD1ka54Ln3MAVxB1Tp4pt0ri4qPLmfGKf/HA==<br>
> couldn't get address for '<a href="http://internet-dns3.state.ma.us" target="_blank">internet-dns3.state.ma.us</a>': not found<br>
> couldn't get address for '<a href="http://internet-dns1.state.ma.us" target="_blank">internet-dns1.state.ma.us</a>': not found<br>
> couldn't get address for '<a href="http://internet-dns2.state.ma.us" target="_blank">internet-dns2.state.ma.us</a>': not found<br>
> dig: couldn't get address for '<a href="http://internet-dns3.state.ma.us" target="_blank">internet-dns3.state.ma.us</a>': no more<br>
> [root@myhost data]#<br>
> <br>
> On Tue, Feb 9, 2021 at 10:10 PM Mark Andrews <<a href="mailto:marka@isc.org" target="_blank">marka@isc.org</a>> wrote:<br>
> Well you could try tracing the addresses of the nameservers for which<br>
> there where errors reported. It could be as simple as a routing issue<br>
> between you and these servers.<br>
> <br>
> > On 10 Feb 2021, at 13:25, sami's strat <<a href="mailto:sami.strat@gmail.com" target="_blank">sami.strat@gmail.com</a>> wrote:<br>
> > <br>
> > couldn't get address for '<a href="http://internet-dns1.state.ma.us" target="_blank">internet-dns1.state.ma.us</a>': not found<br>
> > couldn't get address for '<a href="http://internet-dns3.state.ma.us" target="_blank">internet-dns3.state.ma.us</a>': not found<br>
> > couldn't get address for '<a href="http://internet-dns2.state.ma.us" target="_blank">internet-dns2.state.ma.us</a>': not found<br>
> > dig: couldn't get address for '<a href="http://internet-dns1.state.ma.us" target="_blank">internet-dns1.state.ma.us</a>': no more<br>
> <br>
> Yet, I do this on my personal computer at home, and it works without an issue.<br>
> <br>
> Any other thoughts? TIA <br>
<br>
-- <br>
Mark Andrews, ISC<br>
1 Seymour St., Dundas Valley, NSW 2117, Australia<br>
PHONE: +61 2 9871 4742 INTERNET: <a href="mailto:marka@isc.org" target="_blank">
marka@isc.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>