<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
span.pre
{mso-style-name:pre;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I upgraded our DNS servers when the 9.18.28 release came out, and ran into a problem today that I wanted to know if anyone else had seen or had any suggestions about how to debug.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have our DNS configured in a hidden primary configuration, where the primary has internal and external views and serves and internal and external copy of one of our domains. The external version is fairly small, while the internal version
is significantly larger. We use the same DNSSEC keys to sign both versions of the domain. Every once in a while, we have encountered an issue where the unsigned and signed versions of the domain get out of sync, which causes this message to appear in our
logs (note that I have modified all of the following log entries to replace our domain with example.org):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">25-Jul-2024 10:12:32.202 general: error: zone example.org/IN/internal (signed): receive_secure_serial: not exact<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The solution I’ve always been able to follow previously is to comment out the DNSSEC config options in named.conf, restart named with the zone unsigned, retransfer the unsigned zone to our secondaries, and then put back the DNSSEC config
options, restart named, and let it re-sign the zone. It takes a little bit, but normally everything has then gotten back to normal.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Today, however, when I tried to do that, it started to sign the zone – and then named just hung. It stopped updating any of the log files, stopped sending any notifies, and stopped returning DNS data of any sort. When I tried to restart
named via systemctl it had to kill the process because named would not respond. I was able to undo the DNSSEC changes, restart named, and it continued to work. I tried it again, and named hung once again in the middle of signing the zone. Throughout all
of these restarts, the signed version of the external zone continued to work normally.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is frustrating because when named hangs, there are no error messages in the logs that I can see, and no indication of why it has failed. If I try running rndc commands locally I get this error:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">rndc: recv failed: timed out<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Remote servers show a timeout and then I saw this in some of their transfer logs:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">25-Jul-2024 10:27:01.827 general: info: zone example.org/IN: refresh: skipping zone transfer as primary A.B.C.D#53 (source E.F.G.H#0) is unreachable (cached)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I was able to solve that one by sending notifies from the primary after restarting it without DNSSEC, but I really need to get DNSSEC working again.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The configuration for the zone in named.conf is (and yes, I know I need to update to dnssec-policy):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">view "internal" {<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal"> zone "example.org" {<o:p></o:p></p>
<p class="MsoNormal"> type primary;<o:p></o:p></p>
<p class="MsoNormal"> file "/path/to/internal/example.org";<o:p></o:p></p>
<p class="MsoNormal"> key-directory "/path/to/keys";<o:p></o:p></p>
<p class="MsoNormal"> auto-dnssec maintain;<o:p></o:p></p>
<p class="MsoNormal"> inline-signing yes;<o:p></o:p></p>
<p class="MsoNormal"> };<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">};<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does anyone have any suggestions for putting named into a debug mode to try to get more data if it hangs again? I was thinking of turning the DNSSEC options back on but setting “notify no” so it didn’t try to notify the secondaries in
case all of the notifies and zone transfers going on while it was signing was part of the problem.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The memory and CPU resources of the system should be sufficient – it’s got 2 virtual CPUs and 8GB of memory, but it’s not close to using up the memory, and since it doesn’t have clients, the CPU has never been an issue before. I tried
replicating this issue on our test server but it managed to sign the zone with no problems – though it doesn’t have as many clients.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don’t think the new <span class="pre">max-records-per-type</span> or max-types-per-name options are involved as we don’t have any cases where we have that many records with the same name.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Brian<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:Monaco;color:black;mso-ligatures:none">-- </span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Monaco;color:black;mso-ligatures:none">Brian Sebby (he/him/his) | Lead Systems Engineer</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Monaco;color:black;mso-ligatures:none">Email: <a href="mailto:sebby@anl.gov" title="mailto:sebby@anl.gov"><span style="color:#0563C1">sebby@anl.gov</span></a> | Information Technology Infrastructure</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Monaco;color:black;mso-ligatures:none">Phone: +1 630.252.9935 | Business Information Services</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black;mso-ligatures:none"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Monaco;color:black;mso-ligatures:none">Cell: +1 630.921.4305 | Argonne National Laboratory</span><o:p></o:p></p>
</div>
</body>
</html>