[Kea-users] Fwd: Memory leak on dhcp6

Marcin Siodelski marcin at isc.org
Wed May 8 13:51:47 UTC 2019


Nitzan,

I just wanted to let you know that so far we were unable to reproduce
the memory leak with duplicated server-id across the HA pair. Our test
did not include prefix delegation, only IA_NAs. Are your clients
requesting both address assignment and prefix delegation in the single
transaction?

Thanks,

Marcin

On 07/05/2019 10:49, Nitzan Tzelniker wrote:
> Thanks for your support. 
> Unless I miss it in the logs it might also be a good idea to have error
> message if both servers have the same server-id 
> 
> Thanks
> 
> Nitzan
> 
> On Tue, May 7, 2019 at 11:45 AM Marcin Siodelski <marcin at isc.org
> <mailto:marcin at isc.org>> wrote:
> 
>     Hello Nitzan,
> 
>     Thank you for sharing this information. We'll try to reproduce this
>     behavior on our end. If the problem occurs, we'll investigate and fix it
>     because the duplicate server id is surely not a good reason for leaking
>     memory.... ;-)
> 
>     Marcin Siodelski
> 
>     DHCP Software Engineer,
>     ISC
> 
>     On 07/05/2019 10:26, Nitzan Tzelniker wrote:
>     > Hi,
>     >
>     > Just for anybody else that has this issue 
>     > The issue was that I had the same server-id on both servers as we
>     clone
>     > the VM of the first server after it create the server-id
>     > I dont know why it cause memory leak.
>     >
>     > Thenks
>     >
>     > Nitzan
>     >
>     >
>     > On Mon, May 6, 2019 at 6:56 PM Marcin Siodelski <marcin at isc.org
>     <mailto:marcin at isc.org>
>     > <mailto:marcin at isc.org <mailto:marcin at isc.org>>> wrote:
>     >
>     >     Hi Nitzan,
>     >
>     >     We're trying to reproduce this issue using our test tools. If
>     the leak
>     >     is related to the traffic volume processed by the server, it
>     would be
>     >     useful for us to know the average number of packets/second
>     that your
>     >     primary server is receiving.
>     >
>     >     You say that the memory consumption grows to 28% within an
>     hour but I
>     >     don't know how many packets the server has received and
>     processed to
>     >     reach that level. If you can provide the average DHCP traffic
>     rate we
>     >     can craft the test that best mirrors your situation and
>     confirm whether
>     >     or not we see the same thing.
>     >
>     >     Thanks in advance,
>     >
>     >     Marcin Siodelski
>     >     DHCP Software Engineer,
>     >     ISC
>     >
>     >     On 05/05/2019 16:29, Nitzan Tzelniker wrote:
>     >     > Hi,
>     >     >
>     >     > It seems that the issue is related to the HA
>     >     > After I commented the HA part the memory consumption is
>     stayed low 
>     >     > I am adding the relevant configuration if you have any idea
>     >     >
>     >     > Thanks
>     >     >
>     >     > Nitzan
>     >     >
>     >     > Standby server ha configuration part: 
>     >     >
>     >     >     "hooks-libraries": [
>     >     >         {
>     >     >             "library":
>     "/usr/local/lib/hooks/libdhcp_lease_cmds.so",
>     >     >             "parameters": { }
>     >     >         },
>     >     >         {
>     >     >             "library": "/usr/local/lib/hooks/libdhcp_ha.so",
>     >     >             "parameters": {
>     >     >                 "high-availability": [ {
>     >     >                     "this-server-name": "KEA_DHCPv6_2",
>     >     >                     "mode": "hot-standby",
>     >     >                     "heartbeat-delay": 10000,
>     >     >                     "max-response-delay": 20000,
>     >     >                     "max-ack-delay": 10000,
>     >     >                     "max-unacked-clients": 0,
>     >     >                     "peers": [
>     >     >                         {
>     >     >                             "name": "KEA_DHCPv6_1",
>     >     >                             "role": "primary",
>     >     >                             "auto-failover": true
>     >     >                         },
>     >     >                         {
>     >     >                             "name": "KEA_DHCPv6_2",
>     >     >                             "role": "standby",
>     >     >                             "auto-failover": true
>     >     >                         }
>     >     >                     ]
>     >     >                 } ]
>     >     >             }
>     >     >         }
>     >     >     ]
>     >     > }
>     >     >
>     >     > kea-ctrl-agent.conf  from both servers (Only the ip address is
>     >     different ) 
>     >     >
>     >     > {
>     >     >
>     >     > "Control-agent": {
>     >     >     "http-host": "1.1.1.X",
>     >     >     "http-port": 8080,
>     >     >
>     >     >     "control-sockets": {
>     >     >         "dhcp4": {
>     >     >             "socket-type": "unix",
>     >     >             "socket-name": "/tmp/kea-dhcp4-ctrl.sock"
>     >     >         },
>     >     >         "dhcp6": {
>     >     >             "socket-type": "unix",
>     >     >             "socket-name": "/tmp/kea-dhcp6-ctrl.sock"
>     >     >         }
>     >     >     },
>     >     >
>     >     >     "hooks-libraries": [
>     >     >     ]
>     >     > },
>     >     >
>     >     > "Logging":
>     >     > {
>     >     >   "loggers": [
>     >     >     {
>     >     >         "name": "kea-ctrl-agent",
>     >     >         "output_options": [
>     >     >             {
>     >     >                 "output":
>     "/usr/local/var/log/kea-ctrl-agent.log"
>     >     >             }
>     >     >         ],
>     >     >         "severity": "INFO",
>     >     >
>     >     >         "debuglevel": 0
>     >     >     }
>     >     >   ]
>     >     > }
>     >     > }
>     >     > ---------- Forwarded message ---------
>     >     > From: *Nitzan Tzelniker* <nitzan.tzelniker at gmail.com
>     <mailto:nitzan.tzelniker at gmail.com>
>     >     <mailto:nitzan.tzelniker at gmail.com
>     <mailto:nitzan.tzelniker at gmail.com>>
>     >     > <mailto:nitzan.tzelniker at gmail.com
>     <mailto:nitzan.tzelniker at gmail.com>
>     >     <mailto:nitzan.tzelniker at gmail.com
>     <mailto:nitzan.tzelniker at gmail.com>>>>
>     >     > Date: Fri, May 3, 2019 at 10:13 PM
>     >     > Subject: Memory leak on dhcp6
>     >     > To: <kea-users at lists.isc.org
>     <mailto:kea-users at lists.isc.org> <mailto:kea-users at lists.isc.org
>     <mailto:kea-users at lists.isc.org>>
>     >     <mailto:kea-users at lists.isc.org
>     <mailto:kea-users at lists.isc.org> <mailto:kea-users at lists.isc.org
>     <mailto:kea-users at lists.isc.org>>>>
>     >     >
>     >     >
>     >     > Hi,
>     >     >
>     >     > I am running two dhcpv6 server with high-availability
>     >     > We have about 2500 regular leases and 2500 PD leases
>     >     > The setup is working but it look like kea-dhcp6 leak memory 
>     >     > When it start (with all of the leases in the file ) it take less
>     >     then 1%
>     >     > of the memory but after two hours it take 28% and if I will
>     give it 8
>     >     > hours it will crash 
>     >     >
>     >     > I don't see issue on the standby server only on the primary
>     server
>     >     >
>     >     > Anybody saw this behavior  
>     >     > Any idea how to debug it
>     >     >
>     >     >  kea-dhcp6 -V
>     >     > 1.5.0
>     >     > tarball
>     >     > linked with:
>     >     > log4cplus 1.1.3
>     >     > OpenSSL 1.0.2k-fips  26 Jan 2017
>     >     > database:
>     >     > Memfile backend 2.1
>     >     >
>     >     > Running on CentOS 7.6.1810 VM kernel 3.10.0-957.1.3.el7.x86
>     with 2 *
>     >     > vCPU and 2GB RAM 
>     >     >
>     >     > bellow is the dhcp6 config file omitting 102 subsets 
>     >     >
>     >     > {
>     >     > "Dhcp6": {
>     >     >     "interfaces-config": {
>     >     >         "interfaces": [ "ens192/1111:aef:aa77:83::100" ]
>     >     >     },
>     >     >     "mac-sources": [ "docsis-cmts", "client-link-addr-option" ],
>     >     >
>     >     >     "control-socket": {
>     >     >         "socket-type": "unix",
>     >     >         "socket-name": "/tmp/kea-dhcp6-ctrl.sock"
>     >     >     },
>     >     >
>     >     >     "lease-database": {
>     >     >         "type": "memfile",
>     >     >         "lfc-interval": 3600
>     >     >     },
>     >     >     "expired-leases-processing": {
>     >     >         "reclaim-timer-wait-time": 10,
>     >     >         "flush-reclaimed-timer-wait-time": 25,
>     >     >         "hold-reclaimed-time": 3600,
>     >     >         "max-reclaim-leases": 100,
>     >     >         "max-reclaim-time": 250,
>     >     >         "unwarned-reclaim-cycles": 5
>     >     >     },
>     >     >
>     >     >     "renew-timer": 1000,
>     >     >     "rebind-timer": 2000,
>     >     >     "preferred-lifetime": 3000,
>     >     >     "valid-lifetime": 4000,
>     >     >     "option-data": [
>     >     >         {
>     >     >             "name": "dns-servers",
>     >     >             "data": "1111:aef:aa77:1::22, 1111:aef:aa77:2::22"
>     >     >         }
>     >     >     ],
>     >     >     "subnet6": [
>     >     >         {
>     >     >             "subnet": "1111:aec:0:10::/64",
>     >     >             "pools": [ { "pool":
>     >     > "1111:aec:0:10::10-1111:aec:0:10:ffff::" } ],
>     >     >             "pd-pools": [
>     >     >                 {
>     >     >                     "prefix": "1111:aec:10::",
>     >     >                     "prefix-len": 44,
>     >     >                     "delegated-len": 56
>     >     >                 }
>     >     >             ]
>     >     >         },
>     >     > //....
>     >     > //  101 more subnets 
>     >     > //....
>     >     >  "hooks-libraries": [
>     >     >         {
>     >     >             "library": "/usr/local/lib/hooks/libdhcp_ha.so",
>     >     >             "parameters": {
>     >     >                 "high-availability": [ {
>     >     >                     "this-server-name": "KEA_DHCPv6_1",
>     >     >                     "mode": "hot-standby",
>     >     >                     "heartbeat-delay": 10000,
>     >     >                     "max-response-delay": 20000,
>     >     >                     "max-ack-delay": 10000,
>     >     >                     "max-unacked-clients": 0,
>     >     >                     "peers": [
>     >     >                         {
>     >     >                             "name": "KEA_DHCPv6_1",
>     >     >                             "role": "primary",
>     >     >                             "auto-failover": true
>     >     >                         },
>     >     >                         {
>     >     >                             "name": "KEA_DHCPv6_2",
>     >     >                             "role": "standby",
>     >     >                             "auto-failover": true
>     >     >                         }
>     >     >                     ]
>     >     >                 } ]
>     >     >             }
>     >     >         }
>     >     >     ]
>     >     > },
>     >     >
>     >     > "Logging":
>     >     > {
>     >     >   "loggers": [
>     >     >     {
>     >     >         "name": "kea-dhcp6",
>     >     >         "output_options": [
>     >     >             {
>     >     >                 "output": "/usr/local/var/log/kea-dhcp6.log"
>     >     >             }
>     >     >         ],
>     >     >         "severity": "INFO",
>     >     >
>     >     >         "debuglevel": 0
>     >     >     }
>     >     >   ]
>     >     > }
>     >     > }
>     >     > Thanks
>     >     >
>     >     > Nitzan
>     >     >
>     >     > _______________________________________________
>     >     > Kea-users mailing list
>     >     > Kea-users at lists.isc.org <mailto:Kea-users at lists.isc.org>
>     <mailto:Kea-users at lists.isc.org <mailto:Kea-users at lists.isc.org>>
>     >     > https://lists.isc.org/mailman/listinfo/kea-users
>     >     >
>     >
> 



More information about the Kea-users mailing list