Hi list,<div><br></div><div>We have been running a failover pair for years. After upgrading from 4.1.1-P1 to 4.2.1-P1, I noticed the following message in the logs:</div><div><br></div><div> dhcpd: failover: link startup timeout</div>
<div><br></div><div>This happened every 20 seconds, on both servers. An inspection of the leases file showed that both servers were in "communications interrupted" state.</div><div><br></div><div>We had always run both peers on port 520 without issues. I decided to switch to the standard port, 647, just in case. That didn't fix it. I also tried assigning different ports on the primary (647) and the secondary (847). This had no effect either.</div>
<div><br></div><div>A packet capture shows the following TCP sequences: </div><div><br></div><div><div> 7 5.000940 x.x.60.22 -> x.x.32.35 TCP 47015 > dhcp-failover [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TS\</div><div>
V=1574604576 TSER=0 WS=7</div><div> 8 5.001111 x.x.32.35 -> x.x.60.22 TCP dhcp-failover > 47015 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 \</div><div>MSS=1460 TSV=105374765 TSER=1574604576 WS=7</div><div> 9 5.001130 x.x.60.22 -> x.x.32.35 TCP 47015 > dhcp-failover [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=1\</div>
<div>574604577 TSER=105374765</div><div> 10 5.001674 x.x.32.35 -> x.x.60.22 TCP 38331 > dhcp-failover2 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 T\</div><div>SV=105374765 TSER=0 WS=7</div><div> 11 5.001689 x.x.60.22 -> x.x.32.35 TCP dhcp-failover2 > 38331 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0\</div>
<div> MSS=1460 TSV=1574604577 TSER=105374765 WS=7</div><div> 12 5.001842 x.x.32.35 -> x.x.60.22 TCP 38331 > dhcp-failover2 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=\</div><div>105374765 TSER=1574604577</div><div> 13 20.001583 x.x.60.22 -> x.x.32.35 TCP 47015 > dhcp-failover [FIN, ACK] Seq=1 Ack=1 Win=5888 Len=0 \</div>
<div>TSV=1574619577 TSER=105374765</div><div> 14 20.001817 x.x.32.35 -> x.x.60.22 TCP dhcp-failover > 47015 [FIN, ACK] Seq=1 Ack=2 Win=5888 Len=0 \</div><div>TSV=105389765 TSER=1574619577</div><div> 15 20.001833 x.x.60.22 -> x.x.32.35 TCP 47015 > dhcp-failover [ACK] Seq=2 Ack=2 Win=5888 Len=0 TSV=1\</div>
<div>574619577 TSER=105389765</div><div> 16 20.002561 x.x.60.22 -> x.x.32.35 TCP dhcp-failover2 > 38331 [FIN, ACK] Seq=1 Ack=1 Win=5888 Len=0\</div><div> TSV=1574619578 TSER=105374765</div><div> 17 20.002751 x.x.32.35 -> x.x.60.22 TCP 38331 > dhcp-failover2 [FIN, ACK] Seq=1 Ack=2 Win=5888 Len=0\</div>
<div> TSV=105389766 TSER=1574619578</div><div> 18 20.002760 x.x.60.22 -> x.x.32.35 TCP dhcp-failover2 > 38331 [ACK] Seq=2 Ack=2 Win=5888 Len=0 TSV=\</div><div>1574619578 TSER=105389766</div></div><div><br></div><div>
<br></div><div>I double-checked that both servers were running the same dhcpd version (4.2.1-P1) and also that the peer "stanza" was the same on both configurations.</div><div><br></div><div>After running out of ideas, I decided to downgrade back to 4.1.1-P1. After that, the problem was gone. Obviously, something changed between those two versions that is affecting our setup. The release notes mention a few improvements related to failover, but nothing that could explain what we're seeing.</div>
<div><br></div><div>Here are the current peer configuration files for reference:</div><div><br></div><div><div>failover peer "dhcp-peer" {</div><div> primary; # This is the primary server</div>
<div> address x.x.32.35;</div><div> port 647;</div><div> peer address x.x.60.22;</div><div> peer port 847;</div><div> max-response-delay 60;</div><div> max-unacked-updates 10;</div><div> split 128; </div>
<div> mclt 900;</div><div> load balance max seconds 3;</div><div>}</div></div><div><br></div><div><div>failover peer "dhcp-peer" {</div><div> secondary; # This is the secondary server</div>
<div> address x.x.60.22;</div><div> port 847;</div><div> peer address x.x.32.35;</div><div> peer port 647;</div><div> max-response-delay 60;</div><div> max-unacked-updates 10;</div><div> load balance max seconds 3;</div>
<div>}</div></div><div><br></div><div>I also grabbed a trace file using the -tf parameter. Running that with -play, the only failover lines are:</div><div><br></div><div><div>failover peer dhcp-peer: I move from communications-interrupted to startup</div>
<div>failover: listener: no matching state</div><div>failover peer dhcp-peer: I move from startup to communications-interrupted</div><div>failover: link startup timeout</div><div>failover: listener: no matching state</div>
<div>failover: link startup timeout</div><div>failover: listener: no matching state</div><div>failover: link startup timeout</div><div>failover: listener: no matching state</div><div>failover: link startup timeout</div><div>
failover: listener: no matching state</div><div>failover: link startup timeout</div><div>failover: listener: no matching state</div><div>failover: link startup timeout</div><div>failover: listener: no matching state</div>
</div><div><br></div><div>I can provide the trace file and packet captures to ISC if that helps.</div><div><br></div><div>Thank you in advance for any hints.</div><div><br></div><div>Regards,</div><div><br></div><div>Carlos Vicente</div>
<div>University of Oregon</div><div><br></div><div><br></div><div><br></div>