<div dir="ltr">We have client reporting problem resolving <a href="http://www.federalregister.gov/" rel="noreferrer" target="_blank">federalregister.gov</a> for several weeks, which hosted by the two <a href="http://gpo.gov">gpo.gov</a> nameservers, and our nameserver can't resolve anything under <a href="http://gpo.gov">gpo.gov</a>, we have dnssec-validation yes that's all, I really can't think of anything can be done on our end to fix this, but also i don't understand why such generic config like us see such issue not wide-spread? <div><br><div><br><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Mar 14 10:23:32 ipam-dns-in-1 named[3713]: no valid RRSIG resolving '<a href="http://ns3.gpo.gov/DS/IN">ns3.gpo.gov/DS/IN</a>': 162.140.15.100#53</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Mar 14 10:23:32 ipam-dns-in-1 named[3713]: no valid RRSIG resolving '<a href="http://ns3.gpo.gov/DS/IN">ns3.gpo.gov/DS/IN</a>': 162.140.254.200#53</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Mar 14 10:23:32 ipam-dns-in-1 named[3713]: broken trust chain resolving '<a href="http://ns3.gpo.gov/A/IN">ns3.gpo.gov/A/IN</a>': 162.140.15.100#53</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">This is the dumpdb, wonder why pending-answer:</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; glue</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><a href="http://gpo.gov">gpo.gov</a>.<span class="gmail-Apple-converted-space"> </span>85163 <span class="gmail-Apple-converted-space"> </span>NS<span class="gmail-Apple-converted-space"> </span><a href="http://ns3.gpo.gov">ns3.gpo.gov</a>.</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>85163 <span class="gmail-Apple-converted-space"> </span>NS<span class="gmail-Apple-converted-space"> </span><a href="http://ns4.gpo.gov">ns4.gpo.gov</a>.</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; secure</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>2363<span class="gmail-Apple-converted-space"> </span>DS<span class="gmail-Apple-converted-space"> </span>18496 8 1 (</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>20D05E8AF9ED7706AC31146B9E3BEF2D04C4</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>98ED )</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>2363<span class="gmail-Apple-converted-space"> </span>DS<span class="gmail-Apple-converted-space"> </span>18496 8 2 (</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>665A12F38B1E446269351305495D6E5746CD</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>F92D0CBAA34BAF77624C1CDDDD07 )</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; secure</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>2363<span class="gmail-Apple-converted-space"> </span>RRSIG <span class="gmail-Apple-converted-space"> </span>DS 8 2 3600 (</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>20230321224235 20230314224235 24250 gov.</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>ZFdS88EC8WeL8H6jDcylnXBW5FBboNLhI8vT</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>+hU0GFHVDDdhFW5u7qEEtTvyNArbBtZ8xAPA</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>nALlJwe76n1GXEmYtdx/3CPSF/YLZWE9yc+R</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>3eyXyvN65Ht73WtY1qY7cevXcWVxjiZQI7vf</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>bFIS+yCkX4ZXE3U5dS7ydzasO5yIru05hLnD</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>vTb6eHZty1Kb/O+d/v9WtsqgSTPcVXOgaA== )</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; pending-answer</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><a href="http://ns3.gpo.gov">ns3.gpo.gov</a>.<span class="gmail-Apple-converted-space"> </span>889 <span class="gmail-Apple-converted-space"> </span>\-AAAA<span class="gmail-Apple-converted-space"> </span>;-$NXRRSET</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; <a href="http://gpo.gov">gpo.gov</a>. SOA <a href="http://ins1.gpo.gov">ins1.gpo.gov</a>. <a href="http://noc.gpo.gov">noc.gpo.gov</a>. 2010073218 10800 3600 2592000 900</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; glue</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>43567 <span class="gmail-Apple-converted-space"> </span>A <span class="gmail-Apple-converted-space"> </span>162.140.15.100</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; pending-answer</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><a href="http://ns4.gpo.gov">ns4.gpo.gov</a>.<span class="gmail-Apple-converted-space"> </span>889 <span class="gmail-Apple-converted-space"> </span>\-AAAA<span class="gmail-Apple-converted-space"> </span>;-$NXRRSET</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; <a href="http://gpo.gov">gpo.gov</a>. SOA <a href="http://ins1.gpo.gov">ins1.gpo.gov</a>. <a href="http://noc.gpo.gov">noc.gpo.gov</a>. 2010073218 10800 3600 2592000 900</span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">; glue</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>43567 <span class="gmail-Apple-converted-space"> </span>A <span class="gmail-Apple-converted-space"> </span>162.140.254.200</span></p></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 15, 2023 at 1:50 AM Mark Andrews <<a href="mailto:marka@isc.org">marka@isc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> On 15 Mar 2023, at 15:42, Tim Maestas <<a href="mailto:tmaestas95@gmail.com" target="_blank">tmaestas95@gmail.com</a>> wrote:<br>
> <br>
> Named should be sending queries with DO=1 and it should be getting back signed responses. I suspect that you will need to run packet captures of the traffic to and from 162.140.15.100 and 162.140.254.200 port 53 from the nameserver. Either signed responses will cease or DNSSEC requests will cease. In either case having the traffic around the transition should help to determine what is happening.<br>
> <br>
> I've found that, after a fresh restart of named, if I query for "<a href="http://federalregister.gov" rel="noreferrer" target="_blank">federalregister.gov</a> A" I get a good AD response, and then subsequent queries for "<a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>" are successful as well. If however after a restart of named I begin with a query for <a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a> A then I get servfail, and subsequent queries for <a href="http://federealregister.gov" rel="noreferrer" target="_blank">federealregister.gov</a> servfail as well. Here is the tcpdump from the 2nd (failed) case of an initial query for <a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>:<br>
> <br>
> <br>
> reading from file dns.cap, link-type EN10MB (Ethernet), snapshot length 262144<br>
> 04:30:01.114458 IP (tos 0x0, ttl 64, id 35832, offset 0, flags [none], proto UDP (17), length 92)<br>
> 10.0.0.159.43263 > 162.140.254.200.53: [udp sum ok] 15013 [1au] A? <a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>. ar: . OPT UDPsize=512 DO [COOKIE 352538a87bde87a5] (64)<br>
> 04:30:01.204863 IP (tos 0x0, ttl 229, id 4936, offset 0, flags [DF], proto UDP (17), length 80)<br>
> 162.140.254.200.53 > 10.0.0.159.43263: [udp sum ok] 15013*-| q: A? <a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>. 3/0/1 . OPT UDPsize=4096 DO [|domain]<br>
<br>
This is a malformed DNS response. It looks like the server tried to send a truncated response with an OPT record but failed to correctly update the answer count field to zero. <br>
<br>
% dig <a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a> @<a href="http://162.140.15.100" rel="noreferrer" target="_blank">162.140.15.100</a> +dnssec +bufsize=512 +ignore +qr +norec<br>
<br>
; <<>> DiG 9.19.11-dev <<>> www. @<a href="http://162.140.15.100" rel="noreferrer" target="_blank">162.140.15.100</a> +dnssec +bufsize=512 +ignore +qr +norec<br>
;; global options: +cmd<br>
;; Sending:<br>
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57919<br>
;; flags: ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1<br>
<br>
;; OPT PSEUDOSECTION:<br>
; EDNS: version: 0, flags: do; udp: 512<br>
; COOKIE: 4a67cc813cfe03eb<br>
;; QUESTION SECTION:<br>
;<a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>. IN A<br>
<br>
;; QUERY SIZE: 64<br>
<br>
;; Warning: Message parser reports malformed message packet.<br>
;; Got answer:<br>
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57919<br>
;; flags: qr aa tc; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1<br>
<br>
;; QUESTION SECTION:<br>
;<a href="http://www.federalregister.gov" rel="noreferrer" target="_blank">www.federalregister.gov</a>. IN A<br>
<br>
;; ANSWER SECTION:<br>
. 32768 CLASS4096 OPT <br>
;; Query time: 271 msec<br>
;; SERVER: 162.140.15.100#53(162.140.15.100) (UDP)<br>
;; WHEN: Wed Mar 15 16:30:22 AEDT 2023<br>
;; MSG SIZE rcvd: 52<br>
<br>
-- <br>
Mark Andrews, ISC<br>
1 Seymour St., Dundas Valley, NSW 2117, Australia<br>
PHONE: +61 2 9871 4742 INTERNET: <a href="mailto:marka@isc.org" target="_blank">marka@isc.org</a><br>
<br>
</blockquote></div>