<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>Also, adding stork-focused email distro, though I suspect the membership is largely overlapping </span><span style='font-size:11.0pt;font-family:"Segoe UI Emoji",sans-serif'>😊</span><span style='font-size:11.0pt'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Regards<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Marek<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> mxhajduczenia@gmail.com <mxhajduczenia@gmail.com> <br><b>Sent:</b> Tuesday, May 14, 2024 5:55 AM<br><b>To:</b> 'Darren Ankney' <darren.ankney@gmail.com>; 'Kea user's list' <kea-users@lists.isc.org><br><b>Subject:</b> RE: [Kea-users] High memory utilization for Kea server / Stork agent<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt'>Good morning, Darren <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>These were taken from node2, which is a backup node. And no, I have not done anything to the processes, not even restarted them over the course of a last week or so. I looked through the logs and I do not see any explanation for the behavior. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Node1 is where I constantly change the config (it is the lab after all) and the process is restarted over and over again (“service xxx restart” from shell) but even there I would not expect a process hanging for any reason <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>Marek<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Darren Ankney <<a href="mailto:darren.ankney@gmail.com">darren.ankney@gmail.com</a>> <br><b>Sent:</b> Tuesday, May 14, 2024 5:28 AM<br><b>To:</b> <a href="mailto:mxhajduczenia@gmail.com">mxhajduczenia@gmail.com</a>; Kea user's list <<a href="mailto:kea-users@lists.isc.org">kea-users@lists.isc.org</a>><br><b>Subject:</b> Re: [Kea-users] High memory utilization for Kea server / Stork agent<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi Marek,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>It appears that kea-dhcp4. kea-dhcp6, and stork-agent were started multiple times. I do not know how you are starting the daemons but perhaps something changed with the startup process (systemd?) that caused the thing that is starting the daemons to not realize they are running, thusly starting them multiple times?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thank you,<o:p></o:p></p></div><div><p class=MsoNormal>Darren Ankney<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Mon, May 13, 2024 at 2:16<span style='font-family:"Arial",sans-serif'> </span>PM Marek Hajduczenia <<a href="mailto:mxhajduczenia@gmail.com">mxhajduczenia@gmail.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Looking at the historic utilization over the week <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=464 height=335 style='width:4.8333in;height:3.4916in' id="m_-5218776980966642951Picture_x0020_3" src="cid:image001.png@01DAA5CA.0D718050"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>you can clearly see a rather dramatic increase in memory utilization starting 5/10, with no specific changes to what the node is doing. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Marek<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b>From:</b> Marek Hajduczenia <<a href="mailto:mxhajduczenia@gmail.com" target="_blank">mxhajduczenia@gmail.com</a>> <br><b>Sent:</b> Monday, May 13, 2024 12:12 PM<br><b>To:</b> 'Kea user's list' <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>><br><b>Subject:</b> High memory utilization for Kea server / Stork agent <o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Dear colleagues, <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have been observing very high memory utilization for Kea VMs. In ~4 days, the memory utilization has grown from ~2 GB to close to 8GB for no good reason. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=1066 height=78 style='width:11.1083in;height:.8166in' id="m_-5218776980966642951Picture_x0020_1" src="cid:image002.png@01DAA5CA.0D718050"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=366 height=331 style='width:3.8166in;height:3.45in' id="m_-5218776980966642951Picture_x0020_2" src="cid:image003.png@01DAA5CA.0D718050"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I seem to be running multiple copies of processes for some reason, with stork-agent likely being the primary culprit, and Kea DHCP servers being likely the second culprit. I have the very same situation on both lab nodes. I have to restart the server / stork agent periodically to eliminate the memory bloat. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> <span style='color:black;background:yellow'>634 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 7:35.77 /usr/bin/stork-agent</span></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 709 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:18.34 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 710 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:34.27 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 711 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 713 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.01 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 727 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 740 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.63 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 894 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:24.96 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 1921 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.49 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New";color:black;background:yellow'> 15986 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:53.26 /usr/bin/stork-agent</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1340 _kea 20 0 507M 23864 16784 S 0.0 0.3 4:54.19 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1341 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1342 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1343 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1344 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1352 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.43 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1353 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.49 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1354 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.22 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1355 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.25 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 20544 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 20545 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 679 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.04 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 732 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1889 _kea 20 0 507M 19292 14552 S 0.0 0.2 3:05.41 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1890 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1891 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1892 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1893 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1897 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.47 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1898 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.15 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1899 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:10.02 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 1900 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:09.95 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 20586 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.09 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New"'> 20587 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.08 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Regards<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Marek<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal>-- <br>ISC funds the development of this software with paid support subscriptions. Contact us at <a href="https://www.isc.org/contact/" target="_blank">https://www.isc.org/contact/</a> for more information.<br><br>To unsubscribe visit <a href="https://lists.isc.org/mailman/listinfo/kea-users" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a>.<br><br>Kea-users mailing list<br><a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br><a href="https://lists.isc.org/mailman/listinfo/kea-users" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><o:p></o:p></p></div></blockquote></div></div></body></html>