[kea-dev] Performance problems with 1M devices

le trung trunglv2 at vnpt-technology.vn
Wed Mar 29 07:42:50 UTC 2017


Hi Tom,

I'm sorry, but I have difficulties understanding this. Are you saying you send ACK to Kea?
==> sorry , this is mistake, it is DHCP4_PACKET_RECEIVED (DHCPREQUEST (type 3))

What hardware are you running this on? What Kea version are you using?
What lease backend are you using?
==> I installed kea on Vmware Exsi ( 32G RAM, 12 CPUs)  build on physical server (model name      : Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz)
Kea Version:    1.1
Backend: mariaDB

===> Procedure for load testing Kea :  Tool : dhcperf  ,  device --- Layer 2 -- to --- kea . And result as an image I attached.

===> The product  that my company is providing: ONT ( optical network terminal ) 
With Architecture :   ONT  ....  OTL ----  UPE -------- network L3 ------------- Kea server

With  3M subscriber , Could you suggest any ideal for loadbalancing, HA ?



Thank you,
----------------------------------------------------------------------
Trung Le
Trung tâm Giải pháp và Phát triển phần mềm - SSDC
📱 (+84) 918 042404

-----Original Message-----
From: kea-dev [mailto:kea-dev-bounces at lists.isc.org] On Behalf Of Tomek Mrugalski
Sent: Wednesday, March 29, 2017 11:42 AM
To: kea-dev at lists.isc.org
Subject: Re: [kea-dev] Performance problems with 1M devices

On 28/03/17 23:07, le trung wrote:
> I have installed kea-dhcp opensource for providing IP to network 
> device in the my company.
> 
> Kea is work very good with : total of device < 800k device . But when 
> total of device reaches 1M  device , kea seems has a problem:
> 
> -          When so many request come to KEA ( 400 rqs/s )  ( exceed
> request that KEA can process , results of  my load testing for my 
> server is 248 rqs/s). So when KEA sent OFFER(type 2) message to client  
> >>> Client : sent ACK ( type 5) to KEA server.  but  KEA is processing 
> other request (type 2) ( full 248 rqs/s) so  KEA can not more received 
> request  ACK ( type 5). ==è client received IP after  many time sent 
> offer ( affer client sent discover 30 minute or later)
I'm sorry, but I have difficulties understanding this. Are you saying you send ACK to Kea? That doesn't sound right. ACK is a message being sent by the server to clients. If you somehow managed to sent ACK packet to Kea, Kea will ignore it and log an error.

What hardware are you running this on? What Kea version are you using?
What lease backend are you using? We ran performance tests on relatively modern hardware and came up with performance results several times higher than 248 packets/s.

> So I have ideal :  How to configure KEA to process 248 rqs ( offer 
> message )  and drop offer message  come late ( after request 248). 
> Then completed processing total these 248 requests , KEA continutes 
> receive and process the other 248 requests. That mean, Each time KEA 
> just process 248 offer message and drop other reques come.
How exactly is your testing structured? Do you send a bulk of DHCP packets and then wait? Is this an artificial test or actual traffic generated by clients?

Clients have exponential backoff mechanism implemented. After first transmission, the client waits up to 4 seconds (+- 1 second). If the response does not arrive, it will retransmit and roughly double the waiting time (to 7-9 seconds). This will continue until maximum wait time of 64 seconds is reached. If during that time a packet finally comes back, then the client should accept it. The protocol was so designed that if the reply to first transmission gets very long to reach the client for whatever reason, the client should accept it, even though it may have sent additional retransmissions of that packet. Yes, the server may then process those retransmissions, but it doesn't matter for the client as it will have its configuration already applied.

If you *really* want to make kea drop packets after certain number of queued packets, you can tweak your system socket buffers to make them smaller. But I don't believe this will improve the situation much.

What you can do is to investigate why Kea is not able to respond to more than 248 leases/sec. We manage to get around 1100 leases for MySQL and PostgreSQL and around 9000 leases/sec for memfile. Disclaimer: I'm on a business trip right now and quoting those from memory. The numbers may be off a bit.

Maybe you're logging too much? Extensive logging is a sure way to kill performance. If you're using MySQL, make sure it's configured appropriately to handle the load.

If none of this helps, you can consider getting professional, paid support from ISC. See https://www.isc.org/dhcp-subscription/ for details.

Tomek

_______________________________________________
kea-dev mailing list
kea-dev at lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: loadtest kea just  on Layer 2.png
Type: image/png
Size: 67199 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/kea-dev/attachments/20170329/05611522/attachment-0001.png>


More information about the kea-dev mailing list