[kea-dev] Various pull requests

Andrei Pavel andrei.pavel at qualitance.com
Thu Dec 15 16:25:16 UTC 2016


Hi,

We've done a few changes and we've submitted them via pull requests 35 to
47.

There are two dependencies between those that are related to Cassandra.
Cassandra Host Data Source (CHDS) PR contains everything that the Cassandra
update PR has and the stress test PR contains everything that the CHDS PR
contains. In both cases, it might be easier to merge the contained PR first
and then compare the containing PR with it if you so choose.
A bit about each:

*Cassandra update*
We've added some wrapper functions for communicating with Cassandra:
executeWrite for inserts, updates and deletes as they behave similarly,
executeRead for selects. This PR contains other minor changes like coding
style that are really hard to separate. We apologize for the hardship the
code review might entail. It could be minimized by using a formatter on
both codes and then comparing.

*Cassandra host data source*
The database schema is radically different than the MySQL and the
PostgreSQL schemas. Rather than creating a different table for hosts,
reservations, DHCPv4 options and DHCPv6 options respectively, we've
denormalized them all into a single table to benefit from Cassandra's
non-relational nature. To make up for the lack of relations, on insertion,
the reservations and options are matched against hosts on the server and
merged into database entries. When retrieving, each database row is split
into the corresponding host, reservation and options. There can be an
inconsistency in the database due to the order of the changes e.g. if you
insert a host with no reservations and no options followed by the same host
with one reservation will result in 2 entries versus inserting the host
with reservation from the beginning which will result in a single entry. In
spite of this, retrieving the host will give you the attached reservation
in both cases.

Unit tests are exactly like the other HDS with minor adjustments There are
two possible issues that we've found during unit testing.
1. In some unit tests, a few database operations are expected to return the
rows in the order they have been inserted. This cannot be the case with
Cassandra. Since it doesn't support an ORDER BY type of clause on SELECT
statements, results are always ordered by it's internal row UUID or by the
CLUSTERING ORDER BY clause on the INSERT statement. So the order is
consistent, but it cannot be changed at SELECT. We've modified the generic
HDS unittest to sort the results manually if Cassandra HDS passed through
there. We know his is unprecedented in Kea and bad design, but we can't
think of another solution other than removing the sort check altogether.
2. MySQL and PostgreSQL have unique indexes or constraints on the HDS
tables which Cassandra doesn't support. They are half-simulated only
through the primary key that we've chosen for our single table.
So the unique indexes are:

hosts: dhcp_identifier - dhcp_identifier_type - dhcp4_subnet_id
hosts: dhcp_identifier - dhcp_identifier_type - dhcp6_subnet_id
hosts: ipv4_address - dhcp4_subnet_id
ipv6_reservations: address - prefix_len

And our primary key is a hash of concatenated values of the following
columns:

dhcp4_subnet_id - dhcp6_subnet_id - ipv4_address - ipv6_prefix_address -
ipv6_prefix_length - option_code - option_space

So this is like one big loose constraint. It happens to pass the unit tests
that check for duplicate entries, but they aren't thorough enough. So in
one unit test, dhcp_identifier is changed, and constraint #3 is expected to
be triggered. This works for us because we've expressly excluded
dhcp_identifier and type from our key even though it normally should be
part of the row identifier. If constraint #1 was to be checked, you would
change ipv4_address and constraint #3 would be satisfied, but constraint #1
wouldn't, so the INSERT would fail. This doesn't happen in our case. One
costly solution is to do 4 SELECTs (maybe 3) before every insert to check
if the values to be inserted are constrained by existing values in the
database. We'd like to hear your thoughts on this.

*Cassandra host data source stress test*
Some test that we used for benchmarking included as part of the generic HDS
unittest. It does a bunch of inserts followed by select statements, all
timed and repeated on a increasingly number of hosts and. DISABLED_ by
default.

*clang-format*
Added a Clang format style file and a script to apply it to C++ source
files. Very handy if used from inside an IDE since you can apply it to
blocks of code (e.g. your contribution).

*Changes for uniform compilation*
Added #include <config.h> to all .cc files.
Added KEA_CXXFLAGS to Makefiles in order to uniformly compile all libraries.

*docbook-upgrade*
Upgraded Docbook from v4.2 to v5.0
Used script http://docbook.org/xml/5.0/tools/db4-upgrade.xsl as specified
in the installation documentation at http://tdg.docbook.org/tdg/5.0
/appa.html with command:
find . -type f | grep -F ".xml" | xargs -I{} bash -c "xsltproc -o {}
db4-upgrade.xsl {}"

*Doxyfile*
Updated Doxyfile from 1.8.6 to 1.8.11
This is better generated manually with doxygen -g Doxyfile with doxygen
version 1.8.11.

*makefile-am*
report-cpp-coverage wasn't working because of a syntax error
Added coverage target to .NOTPARALLEL to run unit tests and only then build
reports.
Fixed cppcheck.

*googletest*
Corrected gtest library name. Otherwise it doesn't compile --with-gtest.

*initialize-logging-before-reading-conf*
Started logging before reading the "Dhcp4" and "Dhcp6" configuration, but
after reading the logfile location of course, to log errors thrown because
of any configuration loading issues.

*schema-version*
Changed PostgreSQL version variable names to be inline with MySQL and
Cassandra's

*typos*
The important stuff, right? :)
We've run a spell checker and "indentified" some spelling mistakes.
If you'll review from the command line, we recommend one of the following
commands:
*git diff --color-words*
git diff --color-words | sed -n '/31m/,/m/p'
git diff --color-words --unified=0

*with-dhcp*
with-dhcp-mysql -> with-mysql

with-dhcp-pgsql -> with-pgsql

Regards,
Qualitance
-- 
*Andrei Pavel*  |  junior c++ developer  |  QUALITANCE™
*RO:* +40 (748) 220 135 <+40%20748%20220%20135>  |  *office:* +40 (372) 944
741 <+40%20372%20944%20741>
Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest 050556
<https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
andrei.pavel at qualitance.com  |  www.qualitance.com

a Deloitte Technology Fast 50
<http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
 company | ranked in the Inc. 5000 Europe
<http://www.inc.com/inc5000eu/list/2016/> for 2016
--
The information contained in this email is confidential. It is intended
only for the stated addressee and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in another way use or reply to the information contained in
this email. Such unauthorised use may be unlawful. If you have received
this email in error, please notify the sender immediately and delete all
copies from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-dev/attachments/20161215/330a930e/attachment.html>


More information about the kea-dev mailing list