<div dir="ltr">Hi,<div><br></div><div>Just for anybody else that has this issue </div><div>The issue was that I had the same server-id on both servers as we clone the VM of the first server after it create the server-id</div><div>I dont know why it cause memory leak.</div><div><br></div><div>Thenks</div><div><br></div><div>Nitzan</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 6, 2019 at 6:56 PM Marcin Siodelski <<a href="mailto:marcin@isc.org">marcin@isc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Nitzan,<br>
<br>
We're trying to reproduce this issue using our test tools. If the leak<br>
is related to the traffic volume processed by the server, it would be<br>
useful for us to know the average number of packets/second that your<br>
primary server is receiving.<br>
<br>
You say that the memory consumption grows to 28% within an hour but I<br>
don't know how many packets the server has received and processed to<br>
reach that level. If you can provide the average DHCP traffic rate we<br>
can craft the test that best mirrors your situation and confirm whether<br>
or not we see the same thing.<br>
<br>
Thanks in advance,<br>
<br>
Marcin Siodelski<br>
DHCP Software Engineer,<br>
ISC<br>
<br>
On 05/05/2019 16:29, Nitzan Tzelniker wrote:<br>
> Hi,<br>
> <br>
> It seems that the issue is related to the HA<br>
> After I commented the HA part the memory consumption is stayed low <br>
> I am adding the relevant configuration if you have any idea<br>
> <br>
> Thanks<br>
> <br>
> Nitzan<br>
> <br>
> Standby server ha configuration part: <br>
> <br>
>     "hooks-libraries": [<br>
>         {<br>
>             "library": "/usr/local/lib/hooks/libdhcp_lease_cmds.so",<br>
>             "parameters": { }<br>
>         },<br>
>         {<br>
>             "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
>             "parameters": {<br>
>                 "high-availability": [ {<br>
>                     "this-server-name": "KEA_DHCPv6_2",<br>
>                     "mode": "hot-standby",<br>
>                     "heartbeat-delay": 10000,<br>
>                     "max-response-delay": 20000,<br>
>                     "max-ack-delay": 10000,<br>
>                     "max-unacked-clients": 0,<br>
>                     "peers": [<br>
>                         {<br>
>                             "name": "KEA_DHCPv6_1",<br>
>                             "role": "primary",<br>
>                             "auto-failover": true<br>
>                         },<br>
>                         {<br>
>                             "name": "KEA_DHCPv6_2",<br>
>                             "role": "standby",<br>
>                             "auto-failover": true<br>
>                         }<br>
>                     ]<br>
>                 } ]<br>
>             }<br>
>         }<br>
>     ]<br>
> }<br>
> <br>
> kea-ctrl-agent.conf  from both servers (Only the ip address is different ) <br>
> <br>
> {<br>
> <br>
> "Control-agent": {<br>
>     "http-host": "1.1.1.X",<br>
>     "http-port": 8080,<br>
> <br>
>     "control-sockets": {<br>
>         "dhcp4": {<br>
>             "socket-type": "unix",<br>
>             "socket-name": "/tmp/kea-dhcp4-ctrl.sock"<br>
>         },<br>
>         "dhcp6": {<br>
>             "socket-type": "unix",<br>
>             "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
>         }<br>
>     },<br>
> <br>
>     "hooks-libraries": [<br>
>     ]<br>
> },<br>
> <br>
> "Logging":<br>
> {<br>
>   "loggers": [<br>
>     {<br>
>         "name": "kea-ctrl-agent",<br>
>         "output_options": [<br>
>             {<br>
>                 "output": "/usr/local/var/log/kea-ctrl-agent.log"<br>
>             }<br>
>         ],<br>
>         "severity": "INFO",<br>
> <br>
>         "debuglevel": 0<br>
>     }<br>
>   ]<br>
> }<br>
> }<br>
> ---------- Forwarded message ---------<br>
> From: *Nitzan Tzelniker* <<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>>><br>
> Date: Fri, May 3, 2019 at 10:13 PM<br>
> Subject: Memory leak on dhcp6<br>
> To: <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>>><br>
> <br>
> <br>
> Hi,<br>
> <br>
> I am running two dhcpv6 server with high-availability<br>
> We have about 2500 regular leases and 2500 PD leases<br>
> The setup is working but it look like kea-dhcp6 leak memory <br>
> When it start (with all of the leases in the file ) it take less then 1%<br>
> of the memory but after two hours it take 28% and if I will give it 8<br>
> hours it will crash <br>
> <br>
> I don't see issue on the standby server only on the primary server<br>
> <br>
> Anybody saw this behavior  <br>
> Any idea how to debug it<br>
> <br>
>  kea-dhcp6 -V<br>
> 1.5.0<br>
> tarball<br>
> linked with:<br>
> log4cplus 1.1.3<br>
> OpenSSL 1.0.2k-fips  26 Jan 2017<br>
> database:<br>
> Memfile backend 2.1<br>
> <br>
> Running on CentOS 7.6.1810 VM kernel 3.10.0-957.1.3.el7.x86 with 2 *<br>
> vCPU and 2GB RAM <br>
> <br>
> bellow is the dhcp6 config file omitting 102 subsets <br>
> <br>
> {<br>
> "Dhcp6": {<br>
>     "interfaces-config": {<br>
>         "interfaces": [ "ens192/1111:aef:aa77:83::100" ]<br>
>     },<br>
>     "mac-sources": [ "docsis-cmts", "client-link-addr-option" ],<br>
> <br>
>     "control-socket": {<br>
>         "socket-type": "unix",<br>
>         "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
>     },<br>
> <br>
>     "lease-database": {<br>
>         "type": "memfile",<br>
>         "lfc-interval": 3600<br>
>     },<br>
>     "expired-leases-processing": {<br>
>         "reclaim-timer-wait-time": 10,<br>
>         "flush-reclaimed-timer-wait-time": 25,<br>
>         "hold-reclaimed-time": 3600,<br>
>         "max-reclaim-leases": 100,<br>
>         "max-reclaim-time": 250,<br>
>         "unwarned-reclaim-cycles": 5<br>
>     },<br>
> <br>
>     "renew-timer": 1000,<br>
>     "rebind-timer": 2000,<br>
>     "preferred-lifetime": 3000,<br>
>     "valid-lifetime": 4000,<br>
>     "option-data": [<br>
>         {<br>
>             "name": "dns-servers",<br>
>             "data": "1111:aef:aa77:1::22, 1111:aef:aa77:2::22"<br>
>         }<br>
>     ],<br>
>     "subnet6": [<br>
>         {<br>
>             "subnet": "1111:aec:0:10::/64",<br>
>             "pools": [ { "pool":<br>
> "1111:aec:0:10::10-1111:aec:0:10:ffff::" } ],<br>
>             "pd-pools": [<br>
>                 {<br>
>                     "prefix": "1111:aec:10::",<br>
>                     "prefix-len": 44,<br>
>                     "delegated-len": 56<br>
>                 }<br>
>             ]<br>
>         },<br>
> //....<br>
> //  101 more subnets <br>
> //....<br>
>  "hooks-libraries": [<br>
>         {<br>
>             "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
>             "parameters": {<br>
>                 "high-availability": [ {<br>
>                     "this-server-name": "KEA_DHCPv6_1",<br>
>                     "mode": "hot-standby",<br>
>                     "heartbeat-delay": 10000,<br>
>                     "max-response-delay": 20000,<br>
>                     "max-ack-delay": 10000,<br>
>                     "max-unacked-clients": 0,<br>
>                     "peers": [<br>
>                         {<br>
>                             "name": "KEA_DHCPv6_1",<br>
>                             "role": "primary",<br>
>                             "auto-failover": true<br>
>                         },<br>
>                         {<br>
>                             "name": "KEA_DHCPv6_2",<br>
>                             "role": "standby",<br>
>                             "auto-failover": true<br>
>                         }<br>
>                     ]<br>
>                 } ]<br>
>             }<br>
>         }<br>
>     ]<br>
> },<br>
> <br>
> "Logging":<br>
> {<br>
>   "loggers": [<br>
>     {<br>
>         "name": "kea-dhcp6",<br>
>         "output_options": [<br>
>             {<br>
>                 "output": "/usr/local/var/log/kea-dhcp6.log"<br>
>             }<br>
>         ],<br>
>         "severity": "INFO",<br>
> <br>
>         "debuglevel": 0<br>
>     }<br>
>   ]<br>
> }<br>
> }<br>
> Thanks<br>
> <br>
> Nitzan<br>
> <br>
> _______________________________________________<br>
> Kea-users mailing list<br>
> <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br>
> <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
> <br>
<br>
</blockquote></div>