<div dir="ltr">Hi Darren,<br><br>Thank you for your message. <br><br>In fact, we (simply ^^) would like to implement a redundant DHCP cluster with the following characteristics : <br>* A frontend DHCP cluster with 2 KEA instances<br>* A backend GALERA cluster with 4 MariaDB instances<br>* Leases sync / update disabled between the 2 KEA instances for relying only on the database data sync.<br><br>I can confirm that the GALERA cluster works well for other applications without persistent database connections. <div>In this case, we use HAPROXY with a VIP and the requests to MariaDB instances are load balanced between the available database nodes.<br><br>By configuring our KEA instances in such a way, we can confirm that the DHCP service works, but KEA logs show a lot of error messages (which seems to be due to the fact that the database connections don't persist with a load balancer).<br><br>According your comments, we'll try other setups such as :<br>* 2 KEA + 2 MariaDB with HA hook<br>* 4 KEA + 4 MariaDB without HA hook<br><br>In the meantime, we would be delighted if anyone can share a successful implementation of such a cluster.<br><br>Kind regards.</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">Le jeu. 9 janv. 2025 à 17:08, Darren Ankney <<a href="mailto:darren.ankney@gmail.com">darren.ankney@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Sébastien,<br>
<br>
I do not have any familiarity with HAPROXY. However, it seems there<br>
might be a problem between HAPROXY <-> Mariadb from your description<br>
of the behavior. This may not have anything to do with Kea. Are you<br>
able to test with some other software package to see if the same<br>
problem exists? As far as what can be done in Kea to mitigate this<br>
issue, you could setup retries which you've already done.<br>
<br>
As far as the issue<br>
(<a href="https://gitlab.isc.org/isc-projects/kea/-/issues/1746" rel="noreferrer" target="_blank">https://gitlab.isc.org/isc-projects/kea/-/issues/1746</a>) you mentioned<br>
is concerned. This seems to be describing adding support to Kea to<br>
connect to multiple database servers. It does not seem to pertain to<br>
HAPROXY. If Kea supported this, then you wouldn't need HAPROXY as Kea<br>
would be able to be configured with all of your database IP addresses<br>
directly and would somehow choose one to connect to. I think the<br>
comment you mention specifically is a workaround where you run one Kea<br>
server per database node. If the database node fails, that Kea server<br>
would also fail. That is a workable strategy if you were interested<br>
in it. Of course, the HA hook would not be usable if you have more<br>
than two servers. The result of not using the HA hook would be that<br>
all servers would try to answer each client. You would also want to<br>
run the random lease allocator to avoid collisions.<br>
<br>
Thank you,<br>
Darren Ankney<br>
<br>
<br>
On Thu, Jan 9, 2025 at 4:51 AM Sébastien Nauer <<a href="mailto:snauer@hotcity.lu" target="_blank">snauer@hotcity.lu</a>> wrote:<br>
><br>
> Hello Daren,<br>
><br>
> Thank you for your comments.<br>
><br>
> We use MariaDB v11.4.3 which seems to be fully compatible with KEA.<br>
><br>
> In MariaDB logs, we only see such entries :<br>
> > 2025-01-09 9:06:50 8232949 [Warning] Aborted connection 8232949 to db: 'kea' user: 'kea_user' host: '1.2.3.4' real ip: '5.6.7.8' (Got an error reading communication packets)<br>
><br>
> As we don't see such errors by connecting directly a KEA node to a MariaDB node, it seems that the issue only occurs when using HAPROXY in front of the GALERA Cluster.<br>
><br>
> Also, according the following link, it seems that we should have to configure 4 KEA instances (as we use 4 MariaDB instances which each KEA instance being connected to one MariaDB node) ???<a href="https://gitlab.isc.org/isc-projects/kea/-/issues/1746" rel="noreferrer" target="_blank">https://gitlab.isc.org/isc-projects/kea/-/issues/1746</a><br>
><br>
> For troubleshooting this issue, we currently configured HAPROXY with only 1 MariaDB node and source IP stickiness, the aims being :<br>
> * to persist the KEA database connection with the same MariaDB node<br>
> * to be able to connect KEA to another MariaDB node if the current node fails<br>
><br>
> We can confirm that the error occurs in such a simple situation (with only one database node) and we don't understand why KEA generates DATABASE_MYSQL_FATAL_ERROR messages.<br>
><br>
> Could you confirm if we can use HAPROXY in such a manner ?<br>
> Are there specific settings to activate in KEA and/or HAPROXY ?<br>
> Or finaly, is it better not to use HAPROXY in such a scenario ?<br>
><br>
> Any advice for configuring this kind of KEA cluster will be appreciated.<br>
> Kind regards.<br>
><br>
> Le mer. 8 janv. 2025 à 22:59, Darren Ankney <<a href="mailto:darren.ankney@gmail.com" target="_blank">darren.ankney@gmail.com</a>> a écrit :<br>
>><br>
>> Hi Sébastien,<br>
>><br>
>> The frequent disconnections could be anything. Is there anything in the MySQL logs? Also, is this MySQL or MariaDB? MySQL 8 has some performance issues that affect Kea pretty badly (See here: <a href="https://kea.readthedocs.io/en/kea-2.6.1/arm/admin.html#mysql-5-7-vs-mysql-8-vs-mariadb-10-and-11" rel="noreferrer" target="_blank">https://kea.readthedocs.io/en/kea-2.6.1/arm/admin.html#mysql-5-7-vs-mysql-8-vs-mariadb-10-and-11</a>)<br>
>><br>
>> If you haven't already found them, there are a couple of articles about MySQL clusters and Kea:<br>
>> <a href="https://kb.isc.org/docs/getting-started-with-galera-or-percona-for-kea" rel="noreferrer" target="_blank">https://kb.isc.org/docs/getting-started-with-galera-or-percona-for-kea</a><br>
>> <a href="https://kb.isc.org/docs/experimental-mysql-clusters-for-kea" rel="noreferrer" target="_blank">https://kb.isc.org/docs/experimental-mysql-clusters-for-kea</a> (this one might not be relevant)<br>
>><br>
>> Thank you,<br>
>> Darren Ankney<br>
>><br>
>> On Wed, Jan 8, 2025 at 4:51 AM Sébastien Nauer via Kea-users <<a href="mailto:kea-users@lists.isc.org" target="_blank">kea-users@lists.isc.org</a>> wrote:<br>
>>><br>
>>> Hello,<br>
>>><br>
>>> We are experiencing issues with configuring a KEA cluster in high-availability mode with a GALERA Cluster database backend.<br>
>>><br>
>>> As we'd like to use the database cluster for managing leases synchronization :<br>
>>><br>
>>> we configured KEA High Availability mode like this for disabling lease sync & update :<br>
>>><br>
>>> "high-availability": [<br>
>>> {<br>
>>> "this-server-name": "kea01",<br>
>>> "mode": "load-balancing",<br>
>>> "heartbeat-delay": 10000,<br>
>>> "max-response-delay": 10000,<br>
>>> "max-ack-delay": 5000,<br>
>>> "max-unacked-clients": 5,<br>
>>> "send-lease-updates": false,<br>
>>> "sync-leases": false,<br>
>>> "sync-page-limit": 10000,<br>
>>> "sync-timeout": 30000,<br>
>>> "multi-threading": {<br>
>>> "enable-multi-threading": true,<br>
>>> "http-dedicated-listener": true,<br>
>>> "http-listener-threads": 0,<br>
>>> "http-client-threads": 0<br>
>>> },<br>
>>> "peers": [<br>
>>> {<br>
>>> "name": "kea0&",<br>
>>> "url": "<a href="http://1.2.3.4:8080/" rel="noreferrer" target="_blank">http://1.2.3.4:8080/</a>",<br>
>>> "role": "primary",<br>
>>> "auto-failover": true<br>
>>> },<br>
>>> {<br>
>>> "name": "kea04",<br>
>>> "url": "<a href="http://5.6.7.8:8080/" rel="noreferrer" target="_blank">http://5.6.7.8:8080/</a>",<br>
>>> "role": "secondary",<br>
>>> "auto-failover": true<br>
>>> }<br>
>>> ]<br>
>>> }<br>
>>> ]<br>
>>><br>
>>> we configured the lease database like this for connecting to the database cluster :<br>
>>><br>
>>> "lease-database": {<br>
>>> "type": "mysql",<br>
>>> "name": "kea",<br>
>>> "user": "KEA_USER",<br>
>>> "password": "KEA_PASSWORD",<br>
>>> "host": "1.1.1.1",<br>
>>> "max-reconnect-tries" : 10,<br>
>>> "reconnect-wait-time" : 30000<br>
>>> },<br>
>>><br>
>>> we tried to connect to the database backend :<br>
>>><br>
>>> through a VIP with load balancing<br>
>>> through HAPROXY with / without load balancing<br>
>>><br>
>>> When we start KEA service and the servers start, KEA logs show a lot of DATABASE_MYSQL_FATAL_ERROR messages like this :<br>
>>><br>
>>> 2025-01-07 16:18:42.483 ERROR [kea-dhcp4.database/4349.0x34cf8b308500] DATABASE_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable to execute for <SELECT h.host_id, h.dhcp_identifier, h.dhcp_identifier_type, h.dhcp4_subnet_id, h.dhcp6_subnet_id, h.ipv4_address, h.hostname, h.dhcp4_client_classes, h.dhcp6_client_classes, h.user_context, h.dhcp4_next_server, h.dhcp4_server_hostname, h.dhcp4_boot_file_name, h.auth_key, o.option_id, o.code, o.value, o.formatted_value, <a href="http://o.space" rel="noreferrer" target="_blank">o.space</a>, o.persistent, o.cancelled, o.user_context FROM hosts AS h LEFT JOIN dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id = ? AND h.dhcp_identifier_type = ? AND h.dhcp_identifier = ? ORDER BY h.host_id, o.option_id>, reason: Server has gone away (error code: 2006).<br>
>>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000] DHCP4_DB_RECONNECT_LOST_CONNECTION database connection lost.<br>
>>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.hosts/4349.0x34cf8aa12000] DHCPSRV_MYSQL_HOST_DB opening MySQL hosts database: host=10.194.50.12 max-reconnect-tries=10 name=prod_kea password=***** reconnect-wait-time=30000 type=mysql universe=4 user=keaCluster<br>
>>> 2025-01-07 16:18:43.586 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000] DHCP4_DB_RECONNECT_SUCCEEDED database connection recovered.<br>
>>><br>
>>> As we noticed that the database connection is frequently lost for no apparent reason we suspect that there is an issue with using persistent database connections through a VIP with load balancing (maybe related to multithreading also)<br>
>>><br>
>>> Has anyone successfully configured such a setup ? How ?<br>
>>> What do you suggest for avoiding this error message and configuring a high availability DHCP cluster with a high availability database cluster ?<br>
>>><br>
>>> Any feedback from the community will be appreciated !<br>
>>> Kind regards<br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>><br>
>>><br>
>>><br>
>>> Sébastien NAUER| Technical Manager<br>
>>> 11, Avenue Guillaume | L-1651 Luxembourg<br>
>>> Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665<br>
>>> Facebook | Twitter<br>
>>> --<br>
>>> ISC funds the development of this software with paid support subscriptions. Contact us at <a href="https://www.isc.org/contact/" rel="noreferrer" target="_blank">https://www.isc.org/contact/</a> for more information.<br>
>>><br>
>>> To unsubscribe visit <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a>.<br>
>>><br>
>>> Kea-users mailing list<br>
>>> <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br>
>>> <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
><br>
><br>
><br>
> --<br>
><br>
><br>
><br>
> Sébastien NAUER| Technical Manager<br>
> 11, Avenue Guillaume | L-1651 Luxembourg<br>
> Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665<br>
> Facebook | Twitter<br>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><br><table width="320" cellspacing="0" cellpadding="0" style="font-size:12.8px;font-family:Times;margin:0px;padding:0px"><tbody><tr><td width="100" style="width:100px;padding:0px"><img src="cid:ii_15f531873412908f" alt="Inline-Bild 1" width="100" height="100"><br></td><td width="10" style="width:10px"> </td><td valign="top" style="padding:0px"><br><table cellspacing="0" cellpadding="0" style="padding:0px;margin:0px;font-family:"Lucida Grande",sans-serif;font-size:12px;color:rgb(176,176,176);border-collapse:collapse"><tbody><tr style="margin:0px;padding:0px"><td style="padding:0px;white-space:nowrap"><strong><a href="mailto:snauer@hotcity.lu" style="color:rgb(17,85,204);border:none" target="_blank"><span style="color:rgb(3,54,108)">Sébastien NAUER</span></a></strong><span style="color:rgb(224,224,224)">|</span> <span style="color:rgb(85,85,85)">Technical Manager</span></td></tr><tr style="margin:0px;padding:0px;color:rgb(176,164,155)"><td style="padding:0px;white-space:nowrap"><a href="http://goo.gl/maps/GqQcf" style="color:rgb(176,176,176);border:none" target="_blank">11, Avenue Guillaume <span style="color:rgb(224,224,224)">|</span> L-1651 Luxembourg</a></td></tr><tr style="margin:0px;padding:0px;color:rgb(176,164,155)"><td style="font-family:"Lucida Grande",sans-serif;padding:0px;white-space:nowrap"><a style="color:rgb(176,176,176)">Phone: <span style="color:rgb(192,192,192)">(+352)</span> 2663 2660 </a><span style="color:rgb(224,224,224)">|</span> <a style="color:rgb(176,176,176)">Fax: <span style="color:rgb(192,192,192)">(+352)</span> 2663 2665</a></td></tr><tr style="margin:0px;padding:0px;color:rgb(176,164,155)"><td style="font-family:"Lucida Grande",sans-serif;padding:0px;white-space:nowrap"><a href="https://www.facebook.com/hotcity.lu" style="color:rgb(192,192,192);border:none" target="_blank">Facebook</a> <span style="color:rgb(224,224,224)">|</span> <a href="https://twitter.com/hotcity_wifi" style="color:rgb(192,192,192);border:none" target="_blank">Twitter</a></td></tr></tbody></table></td></tr></tbody></table></div></div></div></div></div>