[kea-dev] KEA performance and hook framework
pallotron at fb.com
Mon Mar 14 10:20:30 UTC 2016
Hello KEA developers!
I am writing to discuss about hook framework performance in light of a recent incident we had in our production network.
I am hoping this would be interesting/actionable feedback for you guys.
In short what happened is that a lot of clients (good portion of a cluster of ~10k nodes) went offline and lost their lease; then they came up all at once asking for DHCP leases and our KEA instances were struggling trying to cope with traffic.
Initial symptoms were maxed out RecvQ buffers as seen by `netstat --tunap` command. That was a symptom that the user space application (KEA) was not able to pop packets from that kernel queue fast enough.
We use KEA via an homemade hook library that we use to resolve host information via thrift calls to an internal backend which is relatively slow and has a p99 response time of 80ms.
It seems KEA is a single threaded UDP server using blocking sockets, that means that the next request can’t be processed if the current in-flight one is not done. Some back of the napkin math I did told me that we were doing approx. 12.5 RPS (1000ms / 80ms) as perfdhcp confirmed (see below).
We tried lowering sysctl options (net.core.rmem_default or net.core.rem_max) in the hope that that would at least have the server respond to some request in a reasonable amount of time but it didn’t help.
Anyway, as I said, after the incident was resolved (we had to blackhole traffic and slowly bring up dhcp clients back online), I went running some perfdhcp tests against our KEA instances:
——————— start of perfdhcp —————
[root at dev206.lla1 ~]# ~pallotron/bin/perfdhcp -4 -r 1500 -p 10 -t 1 -b mac=90:e2:ba:95:b9:28 10.88.77.95
Running: perfdhcp -4 -r 1500 -p 10 -t 1 -b mac=90:e2:ba:95:b9:28 10.88.77.95
sent: 1368/14; received: 14/0; drops: 1354/14
sent: 2736/28; received: 28/0; drops: 2708/28
sent: 4101/40; received: 40/0; drops: 4061/40
sent: 5466/56; received: 56/0; drops: 5410/56
sent: 6834/70; received: 70/0; drops: 6764/70
sent: 8198/83; received: 83/0; drops: 8115/83
sent: 9564/97; received: 97/0; drops: 9467/97
sent: 10928/113; received: 113/0; drops: 10815/113
sent: 12290/128; received: 128/0; drops: 12162/128
Rate: 14.3996 exchanges/second, expected rate: 1500 exchanges/second
***Statistics for: DISCOVER-OFFER***
sent packets: 13647
received packets: 144
min delay: 125.167 ms
avg delay: 5072.366 ms
max delay: 9876.990 ms
std deviation: 2869.381 ms
collected packets: 0
***Statistics for: REQUEST-ACK***
sent packets: 144
received packets: 0
Delay summary unavailable! No packets received.
——————— end of perfdhcp —————
We are addressing this problem by modifying our hook library by adding a background thread that makes async backend calls and caches results in memory. This cache would use a low ttl for entries (few seconds), and likely have a fixed size. Hooks would try to fetch from this cache. If there's an entry the hooks pass a result back. If there's not cache entry they queue a backend call and error (basically not respond to the clients). Clients will retry after the cache has been seeded by the background thread and succeed.
However I am wondering if you are considering improving the server performance and improve concurrency for example using some async model for the hook framework.
Happy to discuss, keep up the good work :)
Cluster Infrastructure - Dublin
Pallotron at fb.com
More information about the kea-dev