kea-dhcp4 - benchmarks (memfile and mysql)
nicolas.chaigneau at capgemini.com
Tue Sep 16 14:15:23 UTC 2014
Hello again, and first thanks for the detailed answer.
First about the memfile mode.
I agree the 4100 leases/s I measured is a good value, probably more than enough.
Note that at that rate, I'm running at a bit less than 100% CPU on the core handling the process. I don't experience any packet loss (if I try a higher lease rate/s, then I reach 100% CPU, and at this point things start to get ugly).
For reference, this was with a lease file on RAMDISK, replicated through DRBD in order to ensure there is no data lost in case of hardware failure.
I also tested the memfile with "persist" set to false (disk operations disabled); in this case, I get up to 4900 leases/s.
Again, the limiting factor for me is the CPU; I assume I would get more on a more powerful hardware.
About the possibility of writing the lease file only on shutdown: that may be interesting to some, but that's not a compromise we can settle with. Hardware failures can happen, even sometimes software crash (not to say it would happen with Kea, but we have to assume the worst). We want to be able to recover the leases in such a case.
Thanks for your recommendations for tuning MySQL, I'll look into this.
I'm currently using a regular hard disk, probably a SAS 10k rpm (I'll check).
I don't have a SSD.
I can try to put MySQL data files on RAMDISK.
About postgreSQL: I have no plans to test with postgreSQL for now. (not that it's not a good DB, but time is limited, so unless there are very good reasons to assume it would be better than MySQL, I won't be able to... sorry)
The fact that Kea is not multi-threaded is disappointing.
Our server has 24 cores, 23 of which are hence sitting idle... and only one at almost 100% CPU (when testing the limit on a memfile).
I hope you can add support for multiple cores in the future :) (any idea when this "one day" might be ? I won't hold you to it, this is just to get a rough estimate)
Thanks again, I'll keep you informed of further testing.
> On 12.09.2014 17:25, Chaigneau, Nicolas wrote:
> > I've tried to benchmark Kea 0.9, using perfdhcp.
> > I'm measuring the number of DORA transactions/s the server is able to
> > handle without client experiencing drops (with default perfdhcp
> > drop-time of 1 sec).
> > From Kea Update #3:
> > "Kea now offers reasonably high performance, leases per second wise.
> > We managed to get over 1000 leases/sec for MySQL and 8000 leases/sec
> > for memfile in memory-only mode on one of our high-performance servers.
> > (Your mileage may vary.) >
> > I don't expect the same numbers, since I don't have the same server
> > and setup than you do, but still these are useful comparison values.
> > With a persistent memfile, I get about 4100 leases/sec.
> That's a good data point. I admit that due to other tasks, we haven't run our benchmarks on our own for quite a while. Our last performance results are from time where memfile lacked disk operations. Personally, I'm quite happy with that number.
> > This is about half your number, but still good.
> We had a discussion about providing a way to disable disk writes during normal operation and write it to disk only during shutdown. Obviously that would be rather unsafe, but very helpful if for some reason you need the highest performance possible and are willing to make compromises. One specific real life example where such feature would be useful is if you have a large network that is recovering from a black out or reboot. There are thousands clients hammering your server for an address. So you can switch off disk writes, provision your clients quickly, write the database to disk and switch back to persistent mode.
> > As a comparison, on the same scenario dhcpd (with a single process)
> > handles at most 70 leases/s.
> > With MySQL, though, I do not manage to get anywhere your numbers.
> > I manage to get about 50 leases/s (which is worse than dhcpd...)
> I have 2 comments for this. First is that we found out that you can significantly improve performance by tweaking MySQL parameters. Here's an excerpt of our internal test report (I'm terribly sorry, but I can't publish it in its entirety as it contains customer specific details.)
> The InnoDB engine is used by default by the tables in the Kea's lease database. A number of parameters controlling this this engine can be adjusted to try to improve MySQL performance. The tests outlined in previous sections make it clear that lease storage is one of the critical components affecting the overall server's performance. While there are some areas in the code that can be optimized it is still the case that the server handles over 8000 leases/sec when leases are not stored into the persistent storage (Memfile) and only 1000-1100 leases/sec when the MySQL storage is used. One of the InnoDB variables that has a notable influence on the server's performance is "innodb_flush_log_at_trx_commit". The value of this variable defaults to
> 1 which configures the MySQL log buffer to be flushed for each transaction commit. If the variable is set to 2 the log buffer is written at each transaction commit but the flush operation is not performed; instead, it is performed once a second. In the table below there is a comparison of the maximal achieved rate for these two settings of "innodb_flush_log_at_trx_commit" variable.
> I assume that you use a regular hdd to store your MySQL data. Let's assume that this is a 7200rpm disk, so it does 120 revolutions per second. So the theoretical limit of writes to disk per second is 120.
> But in practice it is typically 60 as you need two writes to disk for each change: the data to a file and a i-node information change. So in general case the number of available write opportunities for your disk is the limiting factor. I haven't made any experiments here, so the following is pure speculation, but in principle the file system may be a factor here, especially if you use journalling one. Tweaking the fs may provide some insights here. Fortunately, with switching innodb_flush_log_at_trx_commit to 2, you can side-step this limitation.
> If you happen to have SSD disk available, it would be very convenient to try that. If even SSD is not enough, you may consider running MySQL or memfile files on a ram-disk. This could be useful for at least two reasons. First, it will eliminate disk operations from the equation, so you'll be able to measure the raw performance of the Kea+DB. Second, if you really need absolute maximum performance, there are battery backed ram drives available, which should give you similar performance.
> As for our performance results, we used a really beefy server. I don't have the exact parameters at hand, but IIRC properly it had a 10000RPM disk with battery-backed, so even in case of power loss it was able to flush its cache. Therefore its flush calls could return immediately.
> That gave us tremendous speed boosts. It had many cores, but in reality only two of them were actively used: one had Kea-dhcp4 process running on it and the other one was running MySQL.
> > I'm using MySQL 5.6.20. I've created the schema using the script
> > provided with the kea 0.9 package.
> That script sets DB engine to InnoDB and does not tweak any other parameters. We chose this was as it is the safest and should work for a typical deployment.
> > I left all MySQL parameters as default.
> > The CPU usage is very low: about 5% for mysqld, and 3% for kea-dhcp4
> > (both processes, as well as the datafiles, are on the same server)
> > Memory isn't an issue either.
> > I've noticed that Kea uses a single database handle.
> > Wouldn't that be a bottleneck ? (and isn't Kea server multi-threaded
> > ?)
> Yes, Kea uses a single connection. It is currently, single-threaded, single-process application. (Caveat: assuming you want to run only
> DHCPv4 without DDNS. DHCPv6 and DDNS are done in separate process).
> We plan to make it multi-process eventually, so you'll be able to run multiple DHCPv4 instances and share your traffic between them. The Kea architecture allows that, but we are not going to implement this in the near future. The reasons is that there are other missing features that will give us better bang for buck, namely host reservations and client classification. Having said that, we know that multi-process capability will be added one day, so every piece of code we write today is written with that future capability in mind.
> On the other hand, we're more than willing to do smaller performance tweaks in the meantime.
> > So my question is, how did you manage to get 1000 leases/s ? am I
> > doing something wrong ? do you have recommandations for the MySQL setup ?
> See above. I'm eager to hear how high you can get with those tips.
> For completeness, I will mention two observations we made. First, we profiled the code using callgrind (part of valgrind tool) and found out that calls to MySQL library took around 35% of the total execution time.
> Second, when we tested memfile without disk operations, we found out that after passing 8000 leases/sec, we start seeing packet drops, even though the CPU load was at around 40% at that time. This means that there was another limiting factor in our setup. We haven't pursued it any further as the 8000 leases/sec was satisfactory for us at that time.
> However, we speculated that it may be something with incoming buffers or interrupt handling.
> Out of curiosity, are you interested in Postgres performance? I haven't managed to benchmark it at all, so we don't know what to expect. In principle, the code uses the same approach as MySQL (prepared statements, binary data), so the performance should be directly related up to the database engine performance.
> Also, would you be willing to repeat memfile tests with disk operations disabled? I'd love to get an independent confirmation of the 8000 leases/sec.
> Finally, I want to honestly admit that Kea developers are not database experts. There may be obvious DB tricks that we missed.
> Hope that helps. Thanks again for testing Kea.
> kea-dev mailing list
> kea-dev at lists.isc.org
This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
More information about the kea-dev