<div dir="ltr"><br><div>Hello Rasmus,</div><div><br></div><div>After about a week or so of time for analysis, it turns out that it was a couple of factors working in concert, for the most part:</div><div><br></div><div>1. lease times were too short (1H), resulting in request storms as entire racks leases would expire roughly simultaneously, swamping the server with requests; I changed the default lease time to 12H, applied some monitoring to keep track, and lease counts recovered within an hour or so and stabilized</div><div><br></div><div>2. some racks timed out when requesting due to distances between the client and the DHCP server due to either simple network distance or packets lost due to asymmetric routing; one of the affected areas is in another segment entirely with different routing and eventually will be firewalled off, so the fastest way to resolve the issue was simply spin up another DHCP server and point the segment's switches IP helpers to it, instead of the original DHCP server. As they're not sharing the same lease/reservation tables, they can get along using the same database without causing conflicts (I think this scenario was actually explicitly tested by the Kea dev team, and found to be stable)</div><div><br></div><div>3. I think the ALLOC_FAIL messages were actually red herring false positives from a rack or racks(s) that haven't yet been assigned scopes, so no leases are available to be granted yet</div><div><br></div><div>Thank you for the feedback -- I was able to work through these issues using insights from your comments, a bit of "rubber duck" debugging with one of our network engineers, and some instrumentation help generated with InfluxDB. :-)</div><div><br></div><div>cheers,</div><div>Klaus</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 5, 2017 at 2:02 AM, Rasmus Edgar <span dir="ltr"><<a href="mailto:regj@arch-ed.dk" target="_blank">regj@arch-ed.dk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:10pt">
<p>Hi Klaus,</p>
<p>I have seen something very similar on vmware with another application receiving a lot udp traffic and unfortunately we never found a solution for it and switched to bare metal as a workaround, which has irked me ever since and I'm interested in finding a root causes for these kinds of problems.</p>
<p>As far as I understand, and according to the netstat man page, Recv-Q is the count of bytes not yet copied by the user program connected to the socket. </p>
<p>Do you have special rules, execute something or do dns lookups when handling dhcp requests?</p>
<p>Have you read the comments on ALLOC_ENGINE_V4_ALLOC_FAIL?</p>
<p>"% ALLOC_ENGINE_V4_ALLOC_FAIL %1: failed to allocate an IPv4 address after %2 attempt(s)<br>The DHCP allocation engine gave up trying to allocate an IPv4 address<br>after the specified number of attempts. This probably means that the<br>address pool from which the allocation is being attempted is either<br>empty, or very nearly empty. As a result, the client will have been<br>refused a lease. The first argument includes the client identification<br>information.<br><br>This message may indicate that your address pool is too small for the<br>number of clients you are trying to service and should be expanded.<br>Alternatively, if the you know that the number of concurrently active<br>clients is less than the addresses you have available, you may want to<br>consider reducing the lease lifetime. In this way, addresses allocated<br>to clients that are no longer active on the network will become available<br>sooner."</p>
<p>Br,</p>
<p>Rasmus</p><div><div class="h5">
<p>Klaus Steden skrev den 2017-10-05 03:03:</p>
</div></div><blockquote type="cite" style="padding:0 0.4em;border-left:#1010ff 2px solid;margin:0"><div><div class="h5">
<div dir="ltr"><br>
<div>Hi everyone,</div>
<div> </div>
<div>We've been using Kea successfully for several months now as a key part of our provisioning process. However, it seems like the server we're running it on (a VM running under XenServer 6.5) isn't beefy enough, but I'm not 100% confident in that diagnosis.</div>
<div> </div>
<div>There are currently ~200 unique subnets defined, about 2/3rd of which are use to provide a single lease during provisioning, at which point the host in question assigns itself a static IP. There are 77 subnets that are actively in use (for IPMI), with the following lease attributes:</div>
<div>
<p class="m_-4043283279264654126gmail-p1"><span class="m_-4043283279264654126gmail-s1"><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>"valid-lifetime": 4000,<br></span><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>"renew-timer": 1000,<br><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>"rebind-timer": 2000,<br><br>From what I'm seeing in the output of tcpdump, there are a LOT more requests coming in than replies going out, and <em>netstat</em> seems to confirm that:<br><br></p>
<p class="m_-4043283279264654126gmail-p1"><span class="m_-4043283279264654126gmail-s1"># netstat -us<br></span>...<br>Udp:<br><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>71774 packets received<br><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>100 packets to unknown port received.<br><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>565 packet receive errors<br><span class="m_-4043283279264654126gmail-Apple-converted-space"> </span>4911 packets sent</p>
<p class="m_-4043283279264654126gmail-p1">If I monitor <em>netstat</em> continuously, I see spikes on the RecvQ for Kea that fluctuate wildly, anywhere between 0 and nearly 500K (and sometimes higher) moment to moment.</p>
<p class="m_-4043283279264654126gmail-p1">The log also reports a lot of ALLOC_ENGINE_V4_ALLOC_FAIL errors after typically 53 attempts (not sure why 53, but that number seems to be the typical upper limit before failure is confirmed).</p>
<p class="m_-4043283279264654126gmail-p1">I've been experimenting over the last hour or so with tuning various kernel parameters (net.ip4.udp_mem, net.core.rmem_default, net.core.netdev_max_backlog, etc.) but those don't appear to make any kind of difference, and the RecvQ remains high.</p>
<p class="m_-4043283279264654126gmail-p1">Is there any way I can either tune the daemon to handle this kind of backlog, or a list of which kernel tuneables I should be looking at modifying? Is there a more clear way to determine if I've got a genuine performance limitation that we're just now running into?</p>
<p class="m_-4043283279264654126gmail-p1">I've got a bare metal machine temporarily helping carry the burden and it doesn't have these issues, but then again, it's not carrying the full load; I'm loath to dedicate a whole physical server just to DHCP, but if the load is going to remain high like this, maybe that's just what I have to do.</p>
<p class="m_-4043283279264654126gmail-p1">thanks,<br>Klaus</p>
</div>
</div>
<br>
</div></div><div class="m_-4043283279264654126pre" style="margin:0;padding:0;font-family:monospace">______________________________<wbr>_________________<br> Kea-users mailing list<br> <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br> <a href="https://lists.isc.org/mailman/listinfo/kea-users" target="_blank">https://lists.isc.org/mailman/<wbr>listinfo/kea-users</a></div>
</blockquote>
<p><br></p>
</div>
</blockquote></div><br></div>