kea-dhcp4 - benchmarks (memfile and mysql)

Tomek Mrugalski tomasz at isc.org
Tue Sep 16 10:58:14 UTC 2014


On 12.09.2014 17:25, Chaigneau, Nicolas wrote:
> I’ve tried to benchmark Kea 0.9, using perfdhcp.
Great!

> I’m measuring the number of DORA transactions/s the server is able to
> handle without client experiencing drops (with default perfdhcp
> drop-time of 1 sec).
> 
> From Kea Update #3:
> 
> “Kea now offers reasonably high performance, leases per second wise. We
> managed to get over 1000 leases/sec for MySQL and 8000 leases/sec for
> memfile in memory-only mode on one of our high-performance servers.
> (Your mileage may vary.) »
> 
> I don’t expect the same numbers, since I don’t have the same server and
> setup than you do, but still these are useful comparison values.
> 
> With a persistent memfile, I get about 4100 leases/sec.
That's a good data point. I admit that due to other tasks, we haven't
run our benchmarks on our own for quite a while. Our last performance
results are from time where memfile lacked disk operations. Personally,
I'm quite happy with that number.

> This is about half your number, but still good.
We had a discussion about providing a way to disable disk writes during
normal operation and write it to disk only during shutdown. Obviously
that would be rather unsafe, but very helpful if for some reason you
need the highest performance possible and are willing to make
compromises. One specific real life example where such feature would be
useful is if you have a large network that is recovering from a black
out or reboot. There are thousands clients hammering your server for an
address. So you can switch off disk writes, provision your clients
quickly, write the database to disk and switch back to persistent mode.

> As a comparison, on the same scenario dhcpd (with a single process)
> handles at most 70 leases/s.
:)

> With MySQL, though, I do not manage to get anywhere your numbers.
> 
> I manage to get about 50 leases/s (which is worse than dhcpd…)
I have 2 comments for this. First is that we found out that you can
significantly improve performance by tweaking MySQL parameters. Here's
an excerpt of our internal test report (I'm terribly sorry, but I can't
publish it in its entirety as it contains customer specific details.)

innodb_flush_log_at_trx_commit
The InnoDB engine is used by default by the tables in the Kea's lease
database. A number of parameters controlling this this engine can be
adjusted to try to improve MySQL performance. The tests outlined in
previous sections make it clear that lease storage is one of the
critical components affecting the overall server's performance. While
there are some areas in the code that can be optimized it is still the
case that the server handles over 8000 leases/sec when leases are not
stored into the persistent storage (Memfile) and only 1000-1100
leases/sec when the MySQL storage is used. One of the InnoDB variables
that has a notable influence on the server's performance is
"innodb_flush_log_at_trx_commit". The value of this variable defaults to
1 which configures the MySQL log buffer to be flushed for each
transaction commit. If the variable is set to 2 the log buffer is
written at each transaction commit but the flush operation is not
performed; instead, it is performed once a second. In the table below
there is a comparison of the maximal achieved rate for these two
settings of "innodb_flush_log_at_trx_commit" variable.

I assume that you use a regular hdd to store your MySQL data. Let's
assume that this is a 7200rpm disk, so it does 120 revolutions per
second. So the theoretical limit of writes to disk per second is 120.
But in practice it is typically 60 as you need two writes to disk for
each change: the data to a file and a i-node information change. So in
general case the number of available write opportunities for your disk
is the limiting factor. I haven't made any experiments here, so the
following is pure speculation, but in principle the file system may be a
factor here, especially if you use journalling one. Tweaking the fs may
provide some insights here. Fortunately, with switching
innodb_flush_log_at_trx_commit to 2, you can side-step this limitation.

If you happen to have SSD disk available, it would be very convenient to
try that. If even SSD is not enough, you may consider running MySQL or
memfile files on a ram-disk. This could be useful for at least two
reasons. First, it will eliminate disk operations from the equation, so
you'll be able to measure the raw performance of the Kea+DB. Second, if
you really need absolute maximum performance, there are battery backed
ram drives available, which should give you similar performance.

As for our performance results, we used a really beefy server. I don't
have the exact parameters at hand, but IIRC properly it had a 10000RPM
disk with battery-backed, so even in case of power loss it was able to
flush its cache. Therefore its flush calls could return immediately.
That gave us tremendous speed boosts. It had many cores, but in reality
only two of them were actively used: one had Kea-dhcp4 process running
on it and the other one was running MySQL.

> I’m using MySQL 5.6.20. I’ve created the schema using the script
> provided with the kea 0.9 package.
That script sets DB engine to InnoDB and does not tweak any other
parameters. We chose this was as it is the safest and should work for a
typical deployment.

> I left all MySQL parameters as default.
> 
> The CPU usage is very low: about 5% for mysqld, and 3% for kea-dhcp4
> (both processes, as well as the datafiles, are on the same server)
> 
> Memory isn’t an issue either.
> 
> I’ve noticed that Kea uses a single database handle.
> 
> Wouldn’t that be a bottleneck ? (and isn’t Kea server multi-threaded ?)
Yes, Kea uses a single connection. It is currently, single-threaded,
single-process application. (Caveat: assuming you want to run only
DHCPv4 without DDNS. DHCPv6 and DDNS are done in separate process).

We plan to make it multi-process eventually, so you'll be able to run
multiple DHCPv4 instances and share your traffic between them. The Kea
architecture allows that, but we are not going to implement this in the
near future. The reasons is that there are other missing features that
will give us better bang for buck, namely host reservations and client
classification. Having said that, we know that multi-process capability
will be added one day, so every piece of code we write today is written
with that future capability in mind.

On the other hand, we're more than willing to do smaller performance
tweaks in the meantime.

> So my question is, how did you manage to get 1000 leases/s ? am I doing
> something wrong  ? do you have recommandations for the MySQL setup ?
See above. I'm eager to hear how high you can get with those tips.

For completeness, I will mention two observations we made. First, we
profiled the code using callgrind (part of valgrind tool) and found out
that calls to MySQL library took around 35% of the total execution time.

Second, when we tested memfile without disk operations, we found out
that after passing 8000 leases/sec, we start seeing packet drops, even
though the CPU load was at around 40% at that time. This means that
there was another limiting factor in our setup. We haven't pursued it
any further as the 8000 leases/sec was satisfactory for us at that time.
However, we speculated that it may be something with incoming buffers or
interrupt handling.

Out of curiosity, are you interested in Postgres performance? I haven't
managed to benchmark it at all, so we don't know what to expect. In
principle, the code uses the same approach as MySQL (prepared
statements, binary data), so the performance should be directly related
up to the database engine performance.

Also, would you be willing to repeat memfile tests with disk operations
disabled? I'd love to get an independent confirmation of the 8000
leases/sec.

Finally, I want to honestly admit that Kea developers are not database
experts. There may be obvious DB tricks that we missed.

Hope that helps. Thanks again for testing Kea.
Tomek



More information about the kea-dev mailing list