<div dir="ltr">Thanks for your support. <br><div>Unless I miss it in the logs it might also be a good idea to have error message if both servers have the same server-id </div><div><br></div><div>Thanks</div><div><br></div><div>Nitzan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 7, 2019 at 11:45 AM Marcin Siodelski <<a href="mailto:marcin@isc.org">marcin@isc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Nitzan,<br>
<br>
Thank you for sharing this information. We'll try to reproduce this<br>
behavior on our end. If the problem occurs, we'll investigate and fix it<br>
because the duplicate server id is surely not a good reason for leaking<br>
memory.... ;-)<br>
<br>
Marcin Siodelski<br>
<br>
DHCP Software Engineer,<br>
ISC<br>
<br>
On 07/05/2019 10:26, Nitzan Tzelniker wrote:<br>
> Hi,<br>
> <br>
> Just for anybody else that has this issue <br>
> The issue was that I had the same server-id on both servers as we clone<br>
> the VM of the first server after it create the server-id<br>
> I dont know why it cause memory leak.<br>
> <br>
> Thenks<br>
> <br>
> Nitzan<br>
> <br>
> <br>
> On Mon, May 6, 2019 at 6:56 PM Marcin Siodelski <<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a><br>
> <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a>>> wrote:<br>
> <br>
> Hi Nitzan,<br>
> <br>
> We're trying to reproduce this issue using our test tools. If the leak<br>
> is related to the traffic volume processed by the server, it would be<br>
> useful for us to know the average number of packets/second that your<br>
> primary server is receiving.<br>
> <br>
> You say that the memory consumption grows to 28% within an hour but I<br>
> don't know how many packets the server has received and processed to<br>
> reach that level. If you can provide the average DHCP traffic rate we<br>
> can craft the test that best mirrors your situation and confirm whether<br>
> or not we see the same thing.<br>
> <br>
> Thanks in advance,<br>
> <br>
> Marcin Siodelski<br>
> DHCP Software Engineer,<br>
> ISC<br>
> <br>
> On 05/05/2019 16:29, Nitzan Tzelniker wrote:<br>
> > Hi,<br>
> ><br>
> > It seems that the issue is related to the HA<br>
> > After I commented the HA part the memory consumption is stayed low <br>
> > I am adding the relevant configuration if you have any idea<br>
> ><br>
> > Thanks<br>
> ><br>
> > Nitzan<br>
> ><br>
> > Standby server ha configuration part: <br>
> ><br>
> > "hooks-libraries": [<br>
> > {<br>
> > "library": "/usr/local/lib/hooks/libdhcp_lease_cmds.so",<br>
> > "parameters": { }<br>
> > },<br>
> > {<br>
> > "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
> > "parameters": {<br>
> > "high-availability": [ {<br>
> > "this-server-name": "KEA_DHCPv6_2",<br>
> > "mode": "hot-standby",<br>
> > "heartbeat-delay": 10000,<br>
> > "max-response-delay": 20000,<br>
> > "max-ack-delay": 10000,<br>
> > "max-unacked-clients": 0,<br>
> > "peers": [<br>
> > {<br>
> > "name": "KEA_DHCPv6_1",<br>
> > "role": "primary",<br>
> > "auto-failover": true<br>
> > },<br>
> > {<br>
> > "name": "KEA_DHCPv6_2",<br>
> > "role": "standby",<br>
> > "auto-failover": true<br>
> > }<br>
> > ]<br>
> > } ]<br>
> > }<br>
> > }<br>
> > ]<br>
> > }<br>
> ><br>
> > kea-ctrl-agent.conf from both servers (Only the ip address is<br>
> different ) <br>
> ><br>
> > {<br>
> ><br>
> > "Control-agent": {<br>
> > "http-host": "1.1.1.X",<br>
> > "http-port": 8080,<br>
> ><br>
> > "control-sockets": {<br>
> > "dhcp4": {<br>
> > "socket-type": "unix",<br>
> > "socket-name": "/tmp/kea-dhcp4-ctrl.sock"<br>
> > },<br>
> > "dhcp6": {<br>
> > "socket-type": "unix",<br>
> > "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
> > }<br>
> > },<br>
> ><br>
> > "hooks-libraries": [<br>
> > ]<br>
> > },<br>
> ><br>
> > "Logging":<br>
> > {<br>
> > "loggers": [<br>
> > {<br>
> > "name": "kea-ctrl-agent",<br>
> > "output_options": [<br>
> > {<br>
> > "output": "/usr/local/var/log/kea-ctrl-agent.log"<br>
> > }<br>
> > ],<br>
> > "severity": "INFO",<br>
> ><br>
> > "debuglevel": 0<br>
> > }<br>
> > ]<br>
> > }<br>
> > }<br>
> > ---------- Forwarded message ---------<br>
> > From: *Nitzan Tzelniker* <<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>><br>
> > <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a><br>
> <mailto:<a href="mailto:nitzan.tzelniker@gmail.com" target="_blank">nitzan.tzelniker@gmail.com</a>>>><br>
> > Date: Fri, May 3, 2019 at 10:13 PM<br>
> > Subject: Memory leak on dhcp6<br>
> > To: <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>><br>
> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a> <mailto:<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>>>><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> > I am running two dhcpv6 server with high-availability<br>
> > We have about 2500 regular leases and 2500 PD leases<br>
> > The setup is working but it look like kea-dhcp6 leak memory <br>
> > When it start (with all of the leases in the file ) it take less<br>
> then 1%<br>
> > of the memory but after two hours it take 28% and if I will give it 8<br>
> > hours it will crash <br>
> ><br>
> > I don't see issue on the standby server only on the primary server<br>
> ><br>
> > Anybody saw this behavior <br>
> > Any idea how to debug it<br>
> ><br>
> > kea-dhcp6 -V<br>
> > 1.5.0<br>
> > tarball<br>
> > linked with:<br>
> > log4cplus 1.1.3<br>
> > OpenSSL 1.0.2k-fips 26 Jan 2017<br>
> > database:<br>
> > Memfile backend 2.1<br>
> ><br>
> > Running on CentOS 7.6.1810 VM kernel 3.10.0-957.1.3.el7.x86 with 2 *<br>
> > vCPU and 2GB RAM <br>
> ><br>
> > bellow is the dhcp6 config file omitting 102 subsets <br>
> ><br>
> > {<br>
> > "Dhcp6": {<br>
> > "interfaces-config": {<br>
> > "interfaces": [ "ens192/1111:aef:aa77:83::100" ]<br>
> > },<br>
> > "mac-sources": [ "docsis-cmts", "client-link-addr-option" ],<br>
> ><br>
> > "control-socket": {<br>
> > "socket-type": "unix",<br>
> > "socket-name": "/tmp/kea-dhcp6-ctrl.sock"<br>
> > },<br>
> ><br>
> > "lease-database": {<br>
> > "type": "memfile",<br>
> > "lfc-interval": 3600<br>
> > },<br>
> > "expired-leases-processing": {<br>
> > "reclaim-timer-wait-time": 10,<br>
> > "flush-reclaimed-timer-wait-time": 25,<br>
> > "hold-reclaimed-time": 3600,<br>
> > "max-reclaim-leases": 100,<br>
> > "max-reclaim-time": 250,<br>
> > "unwarned-reclaim-cycles": 5<br>
> > },<br>
> ><br>
> > "renew-timer": 1000,<br>
> > "rebind-timer": 2000,<br>
> > "preferred-lifetime": 3000,<br>
> > "valid-lifetime": 4000,<br>
> > "option-data": [<br>
> > {<br>
> > "name": "dns-servers",<br>
> > "data": "1111:aef:aa77:1::22, 1111:aef:aa77:2::22"<br>
> > }<br>
> > ],<br>
> > "subnet6": [<br>
> > {<br>
> > "subnet": "1111:aec:0:10::/64",<br>
> > "pools": [ { "pool":<br>
> > "1111:aec:0:10::10-1111:aec:0:10:ffff::" } ],<br>
> > "pd-pools": [<br>
> > {<br>
> > "prefix": "1111:aec:10::",<br>
> > "prefix-len": 44,<br>
> > "delegated-len": 56<br>
> > }<br>
> > ]<br>
> > },<br>
> > //....<br>
> > // 101 more subnets <br>
> > //....<br>
> > "hooks-libraries": [<br>
> > {<br>
> > "library": "/usr/local/lib/hooks/libdhcp_ha.so",<br>
> > "parameters": {<br>
> > "high-availability": [ {<br>
> > "this-server-name": "KEA_DHCPv6_1",<br>
> > "mode": "hot-standby",<br>
> > "heartbeat-delay": 10000,<br>
> > "max-response-delay": 20000,<br>
> > "max-ack-delay": 10000,<br>
> > "max-unacked-clients": 0,<br>
> > "peers": [<br>
> > {<br>
> > "name": "KEA_DHCPv6_1",<br>
> > "role": "primary",<br>
> > "auto-failover": true<br>
> > },<br>
> > {<br>
> > "name": "KEA_DHCPv6_2",<br>
> > "role": "standby",<br>
> > "auto-failover": true<br>
> > }<br>
> > ]<br>
> > } ]<br>
> > }<br>
> > }<br>
> > ]<br>
> > },<br>
> ><br>
> > "Logging":<br>
> > {<br>
> > "loggers": [<br>
> > {<br>
> > "name": "kea-dhcp6",<br>
> > "output_options": [<br>
> > {<br>
> > "output": "/usr/local/var/log/kea-dhcp6.log"<br>
> > }<br>
> > ],<br>
> > "severity": "INFO",<br>
> ><br>
> > "debuglevel": 0<br>
> > }<br>
> > ]<br>
> > }<br>
> > }<br>
> > Thanks<br>
> ><br>
> > Nitzan<br>
> ><br>
> > _______________________________________________<br>
> > Kea-users mailing list<br>
> > <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a> <mailto:<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a>><br>
> > <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
> ><br>
> <br>
<br>
</blockquote></div>