[kea-dev] Various pull requests

Fri Dec 16 08:46:16 UTC 2016

Forgot to mention that the Cassandra update (PR #35) is meant to replace
"refactored cassandra backend and fixed issues" (PR #26) and was created on
top of it. So only PR #35 should be taken into account and PR #26 should be
closed. If this cannot be easily done e.g. you started working on PR #26
already, let us know how to proceed.

On Thu, Dec 15, 2016 at 6:25 PM, Andrei Pavel <andrei.pavel at qualitance.com>
wrote:

> Hi,
>
> We've done a few changes and we've submitted them via pull requests 35 to
> 47.
>
> There are two dependencies between those that are related to Cassandra.
> Cassandra Host Data Source (CHDS) PR contains everything that the Cassandra
> update PR has and the stress test PR contains everything that the CHDS PR
> contains. In both cases, it might be easier to merge the contained PR first
> and then compare the containing PR with it if you so choose.
> A bit about each:
>
> *Cassandra update*
> We've added some wrapper functions for communicating with Cassandra:
> executeWrite for inserts, updates and deletes as they behave similarly,
> executeRead for selects. This PR contains other minor changes like coding
> style that are really hard to separate. We apologize for the hardship the
> code review might entail. It could be minimized by using a formatter on
> both codes and then comparing.
>
> *Cassandra host data source*
> The database schema is radically different than the MySQL and the
> PostgreSQL schemas. Rather than creating a different table for hosts,
> reservations, DHCPv4 options and DHCPv6 options respectively, we've
> denormalized them all into a single table to benefit from Cassandra's
> non-relational nature. To make up for the lack of relations, on insertion,
> the reservations and options are matched against hosts on the server and
> merged into database entries. When retrieving, each database row is split
> into the corresponding host, reservation and options. There can be an
> inconsistency in the database due to the order of the changes e.g. if you
> insert a host with no reservations and no options followed by the same host
> with one reservation will result in 2 entries versus inserting the host
> with reservation from the beginning which will result in a single entry. In
> spite of this, retrieving the host will give you the attached reservation
> in both cases.
>
> Unit tests are exactly like the other HDS with minor adjustments There are
> two possible issues that we've found during unit testing.
> 1. In some unit tests, a few database operations are expected to return
> the rows in the order they have been inserted. This cannot be the case with
> Cassandra. Since it doesn't support an ORDER BY type of clause on SELECT
> statements, results are always ordered by it's internal row UUID or by the
> CLUSTERING ORDER BY clause on the INSERT statement. So the order is
> consistent, but it cannot be changed at SELECT. We've modified the generic
> HDS unittest to sort the results manually if Cassandra HDS passed through
> there. We know his is unprecedented in Kea and bad design, but we can't
> think of another solution other than removing the sort check altogether.
> 2. MySQL and PostgreSQL have unique indexes or constraints on the HDS
> tables which Cassandra doesn't support. They are half-simulated only
> through the primary key that we've chosen for our single table.
> So the unique indexes are:
>
> hosts: dhcp_identifier - dhcp_identifier_type - dhcp4_subnet_id
> hosts: dhcp_identifier - dhcp_identifier_type - dhcp6_subnet_id
> hosts: ipv4_address - dhcp4_subnet_id
> ipv6_reservations: address - prefix_len
>
> And our primary key is a hash of concatenated values of the following
> columns:
>
> dhcp4_subnet_id - dhcp6_subnet_id - ipv4_address - ipv6_prefix_address -
> ipv6_prefix_length - option_code - option_space
>
> So this is like one big loose constraint. It happens to pass the unit
> tests that check for duplicate entries, but they aren't thorough enough. So
> in one unit test, dhcp_identifier is changed, and constraint #3 is expected
> to be triggered. This works for us because we've expressly excluded
> dhcp_identifier and type from our key even though it normally should be
> part of the row identifier. If constraint #1 was to be checked, you would
> change ipv4_address and constraint #3 would be satisfied, but constraint #1
> wouldn't, so the INSERT would fail. This doesn't happen in our case. One
> costly solution is to do 4 SELECTs (maybe 3) before every insert to check
> if the values to be inserted are constrained by existing values in the
> database. We'd like to hear your thoughts on this.
>
> *Cassandra host data source stress test*
> Some test that we used for benchmarking included as part of the generic
> HDS unittest. It does a bunch of inserts followed by select statements, all
> timed and repeated on a increasingly number of hosts and. DISABLED_ by
> default.
>
> *clang-format*
> Added a Clang format style file and a script to apply it to C++ source
> files. Very handy if used from inside an IDE since you can apply it to
> blocks of code (e.g. your contribution).
>
> *Changes for uniform compilation*
> Added #include <config.h> to all .cc files.
> Added KEA_CXXFLAGS to Makefiles in order to uniformly compile all
> libraries.
>
> *docbook-upgrade*
> Upgraded Docbook from v4.2 to v5.0
> Used script http://docbook.org/xml/5.0/tools/db4-upgrade.xsl as specified
> in the installation documentation at http://tdg.docbook.org/tdg/
> 5.0/appa.html with command:
> find . -type f | grep -F ".xml" | xargs -I{} bash -c "xsltproc -o {}
> db4-upgrade.xsl {}"
>
> *Doxyfile*
> Updated Doxyfile from 1.8.6 to 1.8.11
> This is better generated manually with doxygen -g Doxyfile with doxygen
> version 1.8.11.
>
> *makefile-am*
> report-cpp-coverage wasn't working because of a syntax error
> Added coverage target to .NOTPARALLEL to run unit tests and only then
> build reports.
> Fixed cppcheck.
>
> *googletest*
> Corrected gtest library name. Otherwise it doesn't compile --with-gtest.
>
> *initialize-logging-before-reading-conf*
> Started logging before reading the "Dhcp4" and "Dhcp6" configuration, but
> after reading the logfile location of course, to log errors thrown because
> of any configuration loading issues.
>
> *schema-version*
> Changed PostgreSQL version variable names to be inline with MySQL and
> Cassandra's
>
> *typos*
> The important stuff, right? :)
> We've run a spell checker and "indentified" some spelling mistakes.
> If you'll review from the command line, we recommend one of the following
> commands:
> *git diff --color-words*
> git diff --color-words | sed -n '/31m/,/m/p'
> git diff --color-words --unified=0
>
> *with-dhcp*
> with-dhcp-mysql -> with-mysql
>
> with-dhcp-pgsql -> with-pgsql
>
> Regards,
> Qualitance
> --
> *Andrei Pavel*  |  junior c++ developer  |  QUALITANCE™
> *RO:* +40 (748) 220 135 <+40%20748%20220%20135>  |  *office:* +40 (372)
> 944 741 <+40%20372%20944%20741>
> Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest
> 050556
> <https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
> andrei.pavel at qualitance.com  |  www.qualitance.com
>
> a Deloitte Technology Fast 50
> <http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
>  company | ranked in the Inc. 5000 Europe
> <http://www.inc.com/inc5000eu/list/2016/> for 2016
> --
> The information contained in this email is confidential. It is intended
> only for the stated addressee and access to it by any other person is
> unauthorised. If you are not an addressee, you must not disclose, copy,
> circulate or in another way use or reply to the information contained in
> this email. Such unauthorised use may be unlawful. If you have received
> this email in error, please notify the sender immediately and delete all
> copies from your computer and network.
>

-- 
*Andrei Pavel*  |  junior c++ developer  |  QUALITANCE™
*RO:* +40 (748) 220 135  |  *office:* +40 (372) 944 741
Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest 050556
<https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
andrei.pavel at qualitance.com  |  www.qualitance.com

a Deloitte Technology Fast 50
<http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
 company | ranked in the Inc. 5000 Europe
<http://www.inc.com/inc5000eu/list/2016/> for 2016
--
The information contained in this email is confidential. It is intended
only for the stated addressee and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in another way use or reply to the information contained in
this email. Such unauthorised use may be unlawful. If you have received
this email in error, please notify the sender immediately and delete all
copies from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-dev/attachments/20161216/0726d730/attachment-0001.html>