<div dir="ltr"><div dir="ltr">Hi Marcin,<div><br></div><div>As far as I see in the PCAP most of the customer ask in separate packets but I see few that ask for both in a single packet </div><div>If it will help I took a coredump during the issue <br></div><div><br></div><div>Regards</div><div><br></div><div>Nitzan</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 4:51 PM Marcin Siodelski <<a href="mailto:marcin@isc.org">marcin@isc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Nitzan,<br>
<br>
I just wanted to let you know that so far we were unable to reproduce<br>
the memory leak with duplicated server-id across the HA pair. Our test<br>
did not include prefix delegation, only IA_NAs. Are your clients<br>
requesting both address assignment and prefix delegation in the single<br>
transaction?<br>
<br>
Thanks,<br>
<br>
Marcin<br>
<br>
On 07/05/2019 10:49, Nitzan Tzelniker wrote:<br>
> Thanks for your support. <br>
> Unless I miss it in the logs it might also be a good idea to have error<br>
> message if both servers have the same server-id <br>
> <br>
> Thanks<br>
> <br>
> Nitzan<br>
> <br>
> On Tue, May 7, 2019 at 11:45 AM Marcin Siodelski <<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a><br>
> <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a>>> wrote:<br>
> <br>
> Hello Nitzan,<br>
> <br>
> Thank you for sharing this information. We'll try to reproduce this<br>
> behavior on our end. If the problem occurs, we'll investigate and fix it<br>
> because the duplicate server id is surely not a good reason for leaking<br>
> memory.... ;-)<br>
> <br>
> Marcin Siodelski<br>
> <br>
> DHCP Software Engineer,<br>
> ISC<br>
> <br>
> On 07/05/2019 10:26, Nitzan Tzelniker wrote:<br>
> > Hi,<br>
> ><br>
> > Just for anybody else that has this issue <br>
> > The issue was that I had the same server-id on both servers as we<br>
> clone<br>
> > the VM of the first server after it create the server-id<br>
> > I dont know why it cause memory leak.<br>
> ><br>
> > Thenks<br>
> ><br>
> > Nitzan<br>
> ><br>
> ><br>
> > On Mon, May 6, 2019 at 6:56 PM Marcin Siodelski <<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a><br>
> <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a>><br>
> > <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a> <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a>>>> wrote:<br>
> ><br>
> > Hi Nitzan,<br>
> ><br>
> > We're trying to reproduce this issue using our test tools. If<br>
> the leak<br>
> > is related to the traffic volume processed by the server, it<br>
> would be<br>
> > useful for us to know the average number of packets/second<br>
> that your<br>
> > primary server is receiving.<br>
> ><br>
> > You say that the memory consumption grows to 28% within an<br>
> hour but I<br>
> > don't know how many packets the server has received and<br>
> processed to<br>
> > reach that level. If you can provide the average DHCP traffic<br>
> rate we<br>
> > can craft the test that best mirrors your situation and<br>
> confirm whether<br>
> > or not we see the same thing.<br>
> ><br>
> > Thanks in advance,<br>
> ><br>
> > Marcin Siodelski<br>
> > DHCP Software Engineer,<br>
> > ISC<br>
> ><br>
> > On 05/05/2019 16:29, Nitzan Tzelniker wrote:<br>
> > > Hi,<br>
> > ><br>
> > > It seems that the issue is related to the HA<br>
> > > After I commented the HA part the memory consumption is<br>
> stayed low <br>
> > > I am adding the relevant configuration if you have any idea<br>
> > ><br>
> > > Thanks<br>
> > ><br>
> > > Nitzan<br>
> > ><br>
> > > Standby server ha configuration part: <br>
> > ><br>
> > > "hooks-libraries": [<br>
> > > {<br>
> > > "library":<br>
> "/usr/local/lib/hooks/libdhcp_lease_cmds.so",<br>
> > > "parameters": { }<br>
> > > },<br>
> > > {<br>
> > > "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
> > > "parameters": {<br>
> > > "high-availability": [ {<br>
> > > "this-server-name": "KEA_DHCPv6_2",<br>
> > > "mode": "hot-standby",<br>
> > > "heartbeat-delay": 10000,<br>
> > > "max-response-delay": 20000,<br>
> > > "max-ack-delay": 10000,<br>
> > > "max-unacked-clients": 0,<br>
> > > "peers": [<br>
> > > {<br>
> > > "name": "KEA_DHCPv6_1",<br>
> > > "role": "primary",<br>
> > > "auto-failover": true<br>
> > > },<br>
> > > {<br>
> > > "name": "KEA_DHCPv6_2",<br>
> > > "role": "standby",<br>
> > > "auto-failover": true<br>
> > > }<br>
> > > ]<br>
> > > } ]<br>
> > > }<br>
> > > }<br>
> > > ]<br>
> > > }<br>
> > ><br>
> > > kea-ctrl-agent.conf from both servers (Only the ip address is<br>
> > different ) <br>
> > ><br>
> > > {<br>
> > ><br>
> > > "Control-agent": {<br>
> > > "http-host": "1.1.1.X",<br>
> > > "http-port": 8080,<br>
> > ><br>
> > > "control-sockets": {<br>
> > > "dhcp4": {<br>
> > > "socket-type": "unix",<br>
> > > "socket-name": "/tmp/kea-dhcp4-ctrl.sock"<br>
> > > },<br>
> > > "dhcp6": {<br>
> > > "socket-type": "unix",<br>
> > > "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
> > > }<br>
> > > },<br>
> > ><br>
> > > "hooks-libraries": [<br>
> > > ]<br>
> > > },<br>
> > ><br>
> > > "Logging":<br>
> > > {<br>
> > > "loggers": [<br>
> > > {<br>
> > > "name": "kea-ctrl-agent",<br>
> > > "output_options": [<br>
> > > {<br>
> > > "output":<br>
> "/usr/local/var/log/kea-ctrl-agent.log"<br>
> > > }<br>
> > > ],<br>
> > > "severity": "INFO",<br>
> > ><br>
> > > "debuglevel": 0<br>
> > > }<br>
> > > ]<br>
> > > }<br>
> > > }<br>
> > > ---------- Forwarded message ---------<br>
> > > From: *Nitzan Tzelniker* <<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>><br>
> > <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>>><br>
> > > <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>><br>
> > <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>>>>><br>
> > > Date: Fri, May 3, 2019 at 10:13 PM<br>
> > > Subject: Memory leak on dhcp6<br>
> > > To: <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a><br>
> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a><br>
> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>>><br>
> > <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a><br>
> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a><br>
> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>>>>><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I am running two dhcpv6 server with high-availability<br>
> > > We have about 2500 regular leases and 2500 PD leases<br>
> > > The setup is working but it look like kea-dhcp6 leak memory <br>
> > > When it start (with all of the leases in the file ) it take less<br>
> > then 1%<br>
> > > of the memory but after two hours it take 28% and if I will<br>
> give it 8<br>
> > > hours it will crash <br>
> > ><br>
> > > I don't see issue on the standby server only on the primary<br>
> server<br>
> > ><br>
> > > Anybody saw this behavior <br>
> > > Any idea how to debug it<br>
> > ><br>
> > > kea-dhcp6 -V<br>
> > > 1.5.0<br>
> > > tarball<br>
> > > linked with:<br>
> > > log4cplus 1.1.3<br>
> > > OpenSSL 1.0.2k-fips 26 Jan 2017<br>
> > > database:<br>
> > > Memfile backend 2.1<br>
> > ><br>
> > > Running on CentOS 7.6.1810 VM kernel 3.10.0-957.1.3.el7.x86<br>
> with 2 *<br>
> > > vCPU and 2GB RAM <br>
> > ><br>
> > > bellow is the dhcp6 config file omitting 102 subsets <br>
> > ><br>
> > > {<br>
> > > "Dhcp6": {<br>
> > > "interfaces-config": {<br>
> > > "interfaces": [ "ens192/1111:aef:aa77:83::100" ]<br>
> > > },<br>
> > > "mac-sources": [ "docsis-cmts", "client-link-addr-option" ],<br>
> > ><br>
> > > "control-socket": {<br>
> > > "socket-type": "unix",<br>
> > > "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
> > > },<br>
> > ><br>
> > > "lease-database": {<br>
> > > "type": "memfile",<br>
> > > "lfc-interval": 3600<br>
> > > },<br>
> > > "expired-leases-processing": {<br>
> > > "reclaim-timer-wait-time": 10,<br>
> > > "flush-reclaimed-timer-wait-time": 25,<br>
> > > "hold-reclaimed-time": 3600,<br>
> > > "max-reclaim-leases": 100,<br>
> > > "max-reclaim-time": 250,<br>
> > > "unwarned-reclaim-cycles": 5<br>
> > > },<br>
> > ><br>
> > > "renew-timer": 1000,<br>
> > > "rebind-timer": 2000,<br>
> > > "preferred-lifetime": 3000,<br>
> > > "valid-lifetime": 4000,<br>
> > > "option-data": [<br>
> > > {<br>
> > > "name": "dns-servers",<br>
> > > "data": "1111:aef:aa77:1::22, 1111:aef:aa77:2::22"<br>
> > > }<br>
> > > ],<br>
> > > "subnet6": [<br>
> > > {<br>
> > > "subnet": "1111:aec:0:10::/64",<br>
> > > "pools": [ { "pool":<br>
> > > "1111:aec:0:10::10-1111:aec:0:10:ffff::" } ],<br>
> > > "pd-pools": [<br>
> > > {<br>
> > > "prefix": "1111:aec:10::",<br>
> > > "prefix-len": 44,<br>
> > > "delegated-len": 56<br>
> > > }<br>
> > > ]<br>
> > > },<br>
> > > //....<br>
> > > // 101 more subnets <br>
> > > //....<br>
> > > "hooks-libraries": [<br>
> > > {<br>
> > > "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
> > > "parameters": {<br>
> > > "high-availability": [ {<br>
> > > "this-server-name": "KEA_DHCPv6_1",<br>
> > > "mode": "hot-standby",<br>
> > > "heartbeat-delay": 10000,<br>
> > > "max-response-delay": 20000,<br>
> > > "max-ack-delay": 10000,<br>
> > > "max-unacked-clients": 0,<br>
> > > "peers": [<br>
> > > {<br>
> > > "name": "KEA_DHCPv6_1",<br>
> > > "role": "primary",<br>
> > > "auto-failover": true<br>
> > > },<br>
> > > {<br>
> > > "name": "KEA_DHCPv6_2",<br>
> > > "role": "standby",<br>
> > > "auto-failover": true<br>
> > > }<br>
> > > ]<br>
> > > } ]<br>
> > > }<br>
> > > }<br>
> > > ]<br>
> > > },<br>
> > ><br>
> > > "Logging":<br>
> > > {<br>
> > > "loggers": [<br>
> > > {<br>
> > > "name": "kea-dhcp6",<br>
> > > "output_options": [<br>
> > > {<br>
> > > "output": "/usr/local/var/log/kea-dhcp6.log"<br>
> > > }<br>
> > > ],<br>
> > > "severity": "INFO",<br>
> > ><br>
> > > "debuglevel": 0<br>
> > > }<br>
> > > ]<br>
> > > }<br>
> > > }<br>
> > > Thanks<br>
> > ><br>
> > > Nitzan<br>
> > ><br>
> > > _______________________________________________<br>
> > > Kea-users mailing list<br>
> > > <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a> <mailto:<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a>><br>
> <mailto:<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a> <mailto:<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a>>><br>
> > > <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
> > ><br>
> ><br>
> <br>
<br>
</blockquote></div>