<div dir="ltr">Hi Marek,<div><br></div><div>It appears that kea-dhcp4. kea-dhcp6, and stork-agent were started multiple times. I do not know how you are starting the daemons but perhaps something changed with the startup process (systemd?) that caused the thing that is starting the daemons to not realize they are running, thusly starting them multiple times?</div><div><br></div><div>Thank you,</div><div>Darren Ankney</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 13, 2024 at 2:16 PM Marek Hajduczenia <<a href="mailto:mxhajduczenia@gmail.com">mxhajduczenia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-5218776980966642951"><div lang="EN-US" style="overflow-wrap: break-word;"><div class="m_-5218776980966642951WordSection1"><p class="MsoNormal">Looking at the historic utilization over the week <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span><img width="464" height="335" style="width: 4.8333in; height: 3.4895in;" id="m_-5218776980966642951Picture_x0020_3" src="cid:ii_18f76da9639692e331"></span><u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">you can clearly see a rather dramatic increase in memory utilization starting 5/10, with no specific changes to what the node is doing. <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Marek<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"><p class="MsoNormal"><b><span>From:</span></b><span> Marek Hajduczenia <<a href="mailto:mxhajduczenia@gmail.com" target="_blank">mxhajduczenia@gmail.com</a>> <br><b>Sent:</b> Monday, May 13, 2024 12:12 PM<br><b>To:</b> 'Kea user's list' <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>><br><b>Subject:</b> High memory utilization for Kea server / Stork agent <u></u><u></u></span></p></div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Dear colleagues, <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I have been observing very high memory utilization for Kea VMs. In ~4 days, the memory utilization has grown from ~2 GB to close to 8GB for no good reason. <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span><img width="1066" height="78" style="width: 11.1041in; height: 0.8125in;" id="m_-5218776980966642951Picture_x0020_1" src="cid:ii_18f76da96397745b42"></span><u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span><img width="366" height="331" style="width: 3.8125in; height: 3.4479in;" id="m_-5218776980966642951Picture_x0020_2" src="cid:ii_18f76da9639855d353"></span><u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I seem to be running multiple copies of processes for some reason, with stork-agent likely being the primary culprit, and Kea DHCP servers being likely the second culprit. I have the very same situation on both lab nodes. I have to restart the server / stork agent periodically to eliminate the memory bloat. <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-family:"Courier New""> <span style="background:yellow">634 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 7:35.77 /usr/bin/stork-agent<u></u><u></u></span></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 709 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:18.34 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 710 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:34.27 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 711 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 713 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.01 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 727 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 740 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.63 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 894 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:24.96 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 1921 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.49 /usr/bin/stork-agent<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New";background:yellow"> 15986 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:53.26 /usr/bin/stork-agent</span><span style="font-family:"Courier New""><u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1340 _kea 20 0 507M 23864 16784 S 0.0 0.3 4:54.19 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1341 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1342 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1343 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1344 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1352 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.43 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1353 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.49 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1354 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.22 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1355 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.25 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 20544 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 20545 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 679 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.04 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 732 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1889 _kea 20 0 507M 19292 14552 S 0.0 0.2 3:05.41 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1890 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1891 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1892 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1893 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1897 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.47 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1898 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.15 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1899 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:10.02 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 1900 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:09.95 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 20586 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.09 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Courier New""> 20587 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.08 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf<u></u><u></u></span></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Regards<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Marek<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p></div></div>-- <br>
ISC funds the development of this software with paid support subscriptions. Contact us at <a href="https://www.isc.org/contact/" rel="noreferrer" target="_blank">https://www.isc.org/contact/</a> for more information.<br>
<br>
To unsubscribe visit <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a>.<br>
<br>
Kea-users mailing list<br>
<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br>
<a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
</div></blockquote></div>