[stork-users] [Kea-users] High memory utilization for Kea server / Stork agent
Marek Hajduczenia
mxhajduczenia at gmail.com
Wed May 15 14:12:04 UTC 2024
I will add all the requested details to the https://gitlab.isc.org/isc-projects/stork/-/issues/1389 itself
-----Original Message-----
From: Stork-users <stork-users-bounces at lists.isc.org> On Behalf Of Slawek Figiel
Sent: Wednesday, May 15, 2024 6:54 AM
To: stork-users at lists.isc.org
Subject: Re: [stork-users] [Kea-users] High memory utilization for Kea server / Stork agent
Hi Marek and others!
I've opened https://gitlab.isc.org/isc-projects/stork/-/issues/1389 to analyze if the reported memory usage of the Stork agent process is on the expected level and if it is possible to reduce it.
The separate question is the memory usage of the Kea process. I've no knowledge about the expected memory usage in this application.
Could you provide some details about your configuration? How many subnets, shared networks, host reservations, and client classes do you have configured? Do you use the global or in-subnet reservations? How big are your address/delegated prefix pools? Do you use the config backend or the host database? Do you use HA setup?
I don't need the strict numbers. The order of magnitude would be enough (around 10, 100, 1000, 3000, 5000, 10000, or more than 10000 entries of a particular type?).
Regards,
Slawek
On 14/05/2024 16:21, mxhajduczenia at gmail.com wrote:
> This is what I see on node1 over the last 24 hours – yesterday around
> 10:45 am, I was doing configuration changes on dhcpv6 side, so the
> process would be restarted a few times, it would run, ramping up
> memory utilization is seems, and then got restarted again and has been
> running since. No changes were made to dhcpv4 process or stork agent
>
> I am happy to share the config file(s) with you to see whether there
> is any issue with them and any potential ways of improving memory utilization.
>
> Regards
>
> Marek
>
> *From:*mxhajduczenia at gmail.com <mxhajduczenia at gmail.com>
> *Sent:* Tuesday, May 14, 2024 6:47 AM
> *To:* 'Darren Ankney' <darren.ankney at gmail.com>
> *Cc:* 'Kea user's list' <kea-users at lists.isc.org>;
> stork-users at lists.isc.org
> *Subject:* RE: [Kea-users] High memory utilization for Kea server /
> Stork agent
>
> Also, adding stork-focused email distro, though I suspect the
> membership is largely overlapping 😊
>
> Regards
>
> Marek
>
> *From:*mxhajduczenia at gmail.com <mailto:mxhajduczenia at gmail.com>
> <mxhajduczenia at gmail.com <mailto:mxhajduczenia at gmail.com>>
> *Sent:* Tuesday, May 14, 2024 5:55 AM
> *To:* 'Darren Ankney' <darren.ankney at gmail.com
> <mailto:darren.ankney at gmail.com>>; 'Kea user's list'
> <kea-users at lists.isc.org <mailto:kea-users at lists.isc.org>>
> *Subject:* RE: [Kea-users] High memory utilization for Kea server /
> Stork agent
>
> Good morning, Darren
>
> These were taken from node2, which is a backup node. And no, I have
> not done anything to the processes, not even restarted them over the
> course of a last week or so. I looked through the logs and I do not
> see any explanation for the behavior.
>
> Node1 is where I constantly change the config (it is the lab after
> all) and the process is restarted over and over again (“service xxx restart”
> from shell) but even there I would not expect a process hanging for
> any reason
>
> Marek
>
> *From:*Darren Ankney <darren.ankney at gmail.com
> <mailto:darren.ankney at gmail.com>>
> *Sent:* Tuesday, May 14, 2024 5:28 AM
> *To:* mxhajduczenia at gmail.com <mailto:mxhajduczenia at gmail.com>; Kea
> user's list <kea-users at lists.isc.org <mailto:kea-users at lists.isc.org>>
> *Subject:* Re: [Kea-users] High memory utilization for Kea server /
> Stork agent
>
> Hi Marek,
>
> It appears that kea-dhcp4. kea-dhcp6, and stork-agent were started
> multiple times. I do not know how you are starting the daemons but
> perhaps something changed with the startup process (systemd?) that
> caused the thing that is starting the daemons to not realize they are
> running, thusly starting them multiple times?
>
> Thank you,
>
> Darren Ankney
>
> On Mon, May 13, 2024 at 2:16 PM Marek Hajduczenia
> <mxhajduczenia at gmail.com <mailto:mxhajduczenia at gmail.com>> wrote:
>
> Looking at the historic utilization over the week
>
> you can clearly see a rather dramatic increase in memory utilization
> starting 5/10, with no specific changes to what the node is doing.
>
> Marek
>
> *From:* Marek Hajduczenia <mxhajduczenia at gmail.com
> <mailto:mxhajduczenia at gmail.com>>
> *Sent:* Monday, May 13, 2024 12:12 PM
> *To:* 'Kea user's list' <kea-users at lists.isc.org
> <mailto:kea-users at lists.isc.org>>
> *Subject:* High memory utilization for Kea server / Stork agent
>
> Dear colleagues,
>
> I have been observing very high memory utilization for Kea VMs. In
> ~4 days, the memory utilization has grown from ~2 GB to close to 8GB
> for no good reason.
>
> I seem to be running multiple copies of processes for some reason,
> with stork-agent likely being the primary culprit, and Kea DHCP
> servers being likely the second culprit. I have the very same
> situation on both lab nodes. I have to restart the server / stork
> agent periodically to eliminate the memory bloat.
>
> 634 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 7:35.77
> /usr/bin/stork-agent
>
> 709 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:18.34
> /usr/bin/stork-agent
>
> 710 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:34.27
> /usr/bin/stork-agent
>
> 711 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00
> /usr/bin/stork-agent
>
> 713 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.01
> /usr/bin/stork-agent
>
> 727 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:00.00
> /usr/bin/stork-agent
>
> 740 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.63
> /usr/bin/stork-agent
>
> 894 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:24.96
> /usr/bin/stork-agent
>
> 1921 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 1:29.49
> /usr/bin/stork-agent
>
> 15986 stork-age 20 0 1866M 26240 15824 S 0.0 0.3 0:53.26
> /usr/bin/stork-agent
>
> 1340 _kea 20 0 507M 23864 16784 S 0.0 0.3 4:54.19
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1341 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1342 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1343 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1344 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1352 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.43
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1353 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:05.49
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1354 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.22
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 1355 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:08.25
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 20544 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 20545 _kea 20 0 507M 23864 16784 S 0.0 0.3 0:00.00
> /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
>
> 679 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.04
> /usr/bin/python3
> /usr/share/unattended-upgrades/unattended-upgrade-shutdown
> --wait-for-signal
>
> 732 root 20 0 107M 20736 12596 S 0.0 0.3 0:00.00
> /usr/bin/python3
> /usr/share/unattended-upgrades/unattended-upgrade-shutdown
> --wait-for-signal
>
> 1889 _kea 20 0 507M 19292 14552 S 0.0 0.2 3:05.41
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1890 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1891 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1892 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1893 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.00
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1897 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.47
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1898 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:05.15
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1899 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:10.02
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 1900 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:09.95
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 20586 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.09
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> 20587 _kea 20 0 507M 19292 14552 S 0.0 0.2 0:00.08
> /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>
> Regards
>
> Marek
>
> --
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/
> <https://www.isc.org/contact/> for more information.
>
> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users
> <https://lists.isc.org/mailman/listinfo/kea-users>.
>
> Kea-users mailing list
> Kea-users at lists.isc.org <mailto:Kea-users at lists.isc.org>
> https://lists.isc.org/mailman/listinfo/kea-users
> <https://lists.isc.org/mailman/listinfo/kea-users>
>
>
--
Stork-users mailing list
Stork-users at lists.isc.org
https://lists.isc.org/mailman/listinfo/stork-users
More information about the Stork-users
mailing list