<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Hello Marcin,</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">thank you for your support. I have been confused by docs, sample configs on ISC sites are different.</div><div class="gmail_default"><font face="verdana, sans-serif">Here are timers in milliseconds:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><a href="https://ftp.isc.org/isc/kea/1.4.0-P1/kea-guide.html#high-availability-library">https://ftp.isc.org/isc/kea/1.4.0-P1/kea-guide.html#high-availability-library</a></font><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">and here in seconds:</div><div class="gmail_default"><font face="verdana, sans-serif"><a href="https://kea.isc.org/wiki/HADesign">https://kea.isc.org/wiki/HADesign</a></font><br></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">So I adjusted timers this way:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><div class="gmail_default"> "parameters": {</div><div class="gmail_default"> "high-availability": [ {</div><div class="gmail_default"> "this-server-name": "dhcp-11",</div><div class="gmail_default"> "mode": "load-balancing",</div><div class="gmail_default"> "heartbeat-delay": 2000,</div><div class="gmail_default"> "max-response-delay": 6000,</div><div class="gmail_default"> "max-ack-delay": 5000,</div><div class="gmail_default"> "max-unacked-clients": 0,</div><div><br></div><div>Up to client-class specification, with this configuration server did not sent any responses so I removed this part.</div><div><br></div><div>For subnet selector, it seems to selector work without this specification. I am using relay agents residing in the subnets so there is always option to find match.<br></div><div><br></div><div>regards</div><div>i</div><div><br></div></font></div><br><div class="gmail_quote"><div dir="ltr">št 13. 9. 2018 o 15:31 Marcin Siodelski <<a href="mailto:marcin@isc.org">marcin@isc.org</a>> napísal(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Ivan,<br>
<br>
Thanks for providing the log and config snippets. I have several<br>
comments to share, but neither of them may definitively solve your issue.<br>
<br>
Your heartbeat-delay and max-ack-delay are set to very low values. Note<br>
that they are provided in milliseconds. This means that each server will<br>
be constantly sending heartbeats to its partner and when the partner<br>
doesn't respond to a heartbeat it won't wait long enough for it to<br>
generate DHCP response before it assumes that it is down.<br>
<br>
I realize that you might be doing it to simulate failure scenario where<br>
the surviving server takes over the partner's traffic quickly and the<br>
whole test is not stuck waiting for such transition. However, you may<br>
consider re-running this test with significantly higher values of<br>
heatbeat-delay to make sure that the server being in "partner-down"<br>
state isn't hammered by the heartbeats it needs to generate. It<br>
shouldn't be, but one never knows.<br>
<br>
Secondly, your subnet contains two pools which are not assigned to any<br>
of the HA servers (they lack "client-class" specification). Without this<br>
specification both servers should be able to use both pools. However,<br>
during the normal operation (load balancing) they may end up offering<br>
the same address to two distinct clients and the race condition occurs.<br>
Admittedly, this should not be the reason for the behavior you're<br>
seeing, but I thought I make it clear.<br>
<br>
Thirdly, the subnet configuration provided doesn't contain any subnet<br>
selector. Such selector is typically an "interface" or "relay" parameter<br>
specified at the subnet level. Let's take an "interface" as an example.<br>
If you say "interface": "eth0" in the subnet configuration it means that<br>
the server will assign that subnet for the DHCP traffic received on its<br>
interface "eth0".<br>
<br>
If the subnet selector is not provided the server will try matching<br>
"some" address in the client's packet with available subnets. This can<br>
be: ciaddr, giaddr, source ip address etc. However, if this client is<br>
booting, none of those may be available and the server is unable to<br>
select subnet for the client. As a result it will drop the query.<br>
<br>
However, you say that the servers are responding to the clients before<br>
simulating a failure on one of them. This would mean that the subnet is<br>
selected correctly. However, perhaps the fact that both servers are<br>
online is masking the issue that one of them as actually not responding?<br>
Just a thought.<br>
<br>
Did you try simulating a failure of the other server in the pair? I am<br>
wondering if this is specific to the Kea instance.<br>
<br>
Can you re-run the test with DEBUG logging enabled? We'd see if the<br>
surviving server receives any packets and why it drops them.<br>
<br>
Marcin<br>
<br>
On 13.09.2018 14:09, Ivan Stenda wrote:<br>
> Hello Marcin,<br>
> <br>
> what I see on working host is:<br>
> <br>
> 2018-09-13 13:52:53.079 WARN [kea-dhcp4.ha-hooks/2558]<br>
> HA_LEASE_UPDATE_COMMUNICATIONS_FAILED [hwtype=1 08:3e:5d:10:53:54],<br>
> cid=[no info], tid=0x70b576d2: failed to communicate with dhcp-12<br>
> (<a href="http://10.58.0.12:8080/" rel="noreferrer" target="_blank">http://10.58.0.12:8080/</a>): Connection refused<br>
> 2018-09-13 13:52:53.789 WARN [kea-dhcp4.ha-hooks/2558]<br>
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to dhcp-12<br>
> (<a href="http://10.58.0.12:8080/" rel="noreferrer" target="_blank">http://10.58.0.12:8080/</a>): Connection refused<br>
> 2018-09-13 13:52:54.820 WARN [kea-dhcp4.ha-hooks/2558]<br>
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to dhcp-12<br>
> (<a href="http://10.58.0.12:8080/" rel="noreferrer" target="_blank">http://10.58.0.12:8080/</a>): Connection refused<br>
> 2018-09-13 13:52:55.957 WARN [kea-dhcp4.ha-hooks/2558]<br>
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to dhcp-12<br>
> (<a href="http://10.58.0.12:8080/" rel="noreferrer" target="_blank">http://10.58.0.12:8080/</a>): Connection refused<br>
> 2018-09-13 13:52:55.957 INFO [kea-dhcp4.ha-hooks/2558]<br>
> HA_STATE_TRANSITION server transitions from LOAD-BALANCING to<br>
> PARTNER-DOWN state, partner state is UNDEFINED<br>
> 2018-09-13 13:52:55.957 INFO [kea-dhcp4.ha-hooks/2558]<br>
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner<br>
> while in PARTNER-DOWN state<br>
> <br>
> and can confirm that OFFERs are not send out from working host.<br>
> <br>
> <br>
> configuration snippets here:<br>
> {<br>
> "interfaces-config": {<br>
> "interfaces": [ "ens192" ],<br>
> "dhcp-socket-type": "udp"<br>
> },<br>
> <br>
> {<br>
> "subnet": "<a href="http://10.187.0.0/24" rel="noreferrer" target="_blank">10.187.0.0/24</a> <<a href="http://10.187.0.0/24" rel="noreferrer" target="_blank">http://10.187.0.0/24</a>>",<br>
> "pools": [<br>
> {<br>
> "pool": "10.187.0.10 - 10.187.0.127"<br>
> },<br>
> {<br>
> "pool": "10.187.0.128 - 10.187.0.250"<br>
> }<br>
> ],<br>
> <br>
> "option-data": [<br>
> {<br>
> "name": "routers",<br>
> "data": "10.187.0.1"<br>
> }<br>
> ]<br>
> <br>
> },<br>
> <br>
> "hooks-libraries": [<br>
> {<br>
> "library": "/opt/kea/usr/lib/hooks/libdhcp_lease_cmds.so",<br>
> "parameters": { }<br>
> },<br>
> {<br>
> "library": "/opt/kea/usr/lib/hooks/libdhcp_ha.so",<br>
> "parameters": {<br>
> "high-availability": [ {<br>
> "this-server-name": "dhcp-11",<br>
> "mode": "load-balancing",<br>
> "heartbeat-delay": 10,<br>
> //"max-response-delay": 10000,<br>
> "max-ack-delay": 5,<br>
> "max-unacked-clients": 5,<br>
> "peers": [<br>
> {<br>
> "name": "dhcp-11",<br>
> "url": "<a href="http://10.58.0.11:8080/" rel="noreferrer" target="_blank">http://10.58.0.11:8080/</a>",<br>
> "role": "primary",<br>
> "auto-failover": true<br>
> },<br>
> {<br>
> "name": "dhcp-12",<br>
> "url": "<a href="http://10.58.0.12:8080/" rel="noreferrer" target="_blank">http://10.58.0.12:8080/</a>",<br>
> "role": "secondary",<br>
> "auto-failover": true<br>
> }<br>
> ]<br>
> } ]<br>
> }<br>
> }<br>
> <br>
> ]<br>
> <br>
> },<br>
> <br>
> <br>
> regards<br>
> i<br>
> <br>
> št 13. 9. 2018 o 13:23 Marcin Siodelski <<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a><br>
> <mailto:<a href="mailto:marcin@isc.org" target="_blank">marcin@isc.org</a>>> napísal(a):<br>
> <br>
> On 13.09.2018 09:03, Ivan Stenda wrote:<br>
> > Hello guys,<br>
> ><br>
> > I am trying to set up HA on $subj with no luck. Managed peers to<br>
> talk in<br>
> > between via kea-ctrl-agent, lease updates send from host to host and<br>
> > vice versa but on simulated failure clients are not served seamless.<br>
> > They are doing whole DORA because working host does not send OFFER<br>
> from<br>
> > failed peer pool ...<br>
> ><br>
> > Maybe I am wrong about networking around KEA. Could someone guide<br>
> me for<br>
> > networking setup in case of UDP socket and relay hosts ?<br>
> ><br>
> > regards<br>
> > i<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Kea-users mailing list<br>
> > <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a> <mailto:<a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a>><br>
> > <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
> ><br>
> <br>
> Hello Ivan,<br>
> <br>
> It is hard to say without looking into configurations of both of your HA<br>
> peers.<br>
> <br>
> I guess the first question is whether the server that takes over the<br>
> traffic from the failed partner sends an OFFER (according to your logs)<br>
> and this OFFER doesn't go through the network, or the OFFER is not<br>
> generated by the server. Also, when you're expecting those offers do you<br>
> observe the server which should send this offer being in the<br>
> "partner-down" state (according to logs)?<br>
> <br>
> Marcin Siodelski<br>
> ISC DHCP Engineering<br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Kea-users mailing list<br>
> <a href="mailto:Kea-users@lists.isc.org" target="_blank">Kea-users@lists.isc.org</a><br>
> <a href="https://lists.isc.org/mailman/listinfo/kea-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/kea-users</a><br>
> <br>
<br>
</blockquote></div></div></div></div></div></div>