DHCP relay with UDP source port of 67 causes ISC 3.0.2 to respond with UDP source port of 1

[Frank Bulk]

David:

Yes, when the CMTS was upgraded it related all DHCP traffic a UDP src/dst
port of 67/67.  When the CMTS was running it's original code it relayed with
a UDP src/dst port of 68/67

I agree it, it was very strange that the DHCP server was issuing DHCP
response on UDP src ports other than 67, which is why I was stumped.  

To describe our set up in a little more detail, we're using a third-party
vendor's DHCP box set up in a high-availability configuration (name withheld
to protect the guilty).  Only one appliance runs dhcpd at any one time, and
only that appliance has eth0:0 with the a.b.c.24 IP address applied to it.
When one appliance fails and the other appliance detects it applies a.b.c.24
to its eth0:0 and starts dhcpd.
=================================================
When running:
 Server A  		  Server B
eth0 	eth0:0	eth0	eth0:0	Active server
.22	.24		.23				A

On failover:
 Server A  		  Server B
eth0 	eth0:0	eth0	eth0:0	Active server
.22			.23	.24			B 
=================================================

I should add more detail to the configuration that I've learned since then:
- dhcpd is bound to eth0:0. Changing it to bind to eth0 didn't make any
difference (and shouldn't according to the notes I've read)
- when the CMTS was relaying with a UDP src/dst port of 68/67 about 10 to
20% of the ACKs used a UDP src port of 1.  It might be artificially high
because the clients would just issue another DHCP Inform or Request because
they don't like the DHCP Acks with a UDP src port of 1.  Eventually the
clients' lease would expire and it would renew altogether.
- when the CMTS was upgraded it relayed with a UDP src/dst port of 67/67 and
there were DHCP Offers in addition to Acks with a UDP source port of 1.
- iptables is on each box.  It has a nat POSTROUTING rule to change the src
IP address to match the 'master' IP address.  (We have one DHCP relay in our
that doesn't like it when it issues a DHCP Discover to a.b.c.24 and gets a
response from a.b.c.22 or a.b.c.23, hence the src IP address change.  This
is third-party vendor implemented and supported.)
	-A POSTROUTING -s a.b.c.22 -p udp -m udp --sport 67 -j SNAT
--to-source a.b.c.24
- when I remove the POSTROUTING rule, it's interesting to see that most
everything comes out of the DHCP server with IP source address of a.b.c.22,
as it should, but there are some ACKs with a source address of a.b.c.24 --
and guess what, they all have a src port of 1!  I tried over a dozen
different iptables rules, but no success in catching those aberrant UDP src
port 1 packets and changing them, via iptables, to UDP src port 67.
- this leads me to conjecture that dhcpd, for some of its packets, is not
binding to the right interface, and spewing out an incorrect packet.

I agree, dhcpd shouldn't care what the source port from the DHCP relay, but
it's possible that there's something in the code that's leading dhcpd to
occasionally use a different interface for its output.

Regards,

Frank

P.S. After 30 to 40 hours diagnosing this problem, I was able to use the
'local-address a.b.c.24;' line in dhcpd.conf to resolve this permanently,
which re-confirms to me that it's something with the dhcpd code that it's
occasionally using eth0:0 instead of eth0.

-----Original Message-----
From: dhcp-users-bounce at isc.org [mailto:dhcp-users-bounce at isc.org] On Behalf
Of David W. Hankins
Sent: Thursday, November 02, 2006 6:27 PM
To: dhcp-users at isc.org
Subject: Re: DHCP relay with UDP source port of 67 causes ISC 3.0.2 to
respond with UDP source port of 1

On Sat, Oct 28, 2006 at 01:55:07AM -0500, Frank Bulk wrote:
> When we do that, DHCP Discovers to ISC dhcpd (running on a 3rd-party
> appliance) returns to the CMTS with a DHCP Offer of 1/67 rather than the
> 67/67.  Many Windows client don't seem to like this offer because of the
> source port of 1 and just drop the packet, issuing another DHCP Discover.

The OFFERs came out of the DHCP server with source port 1?  Or
they came out of the CMTS with source port 1?

I think it is highly unlikely that the OFFERs came out of the
DHCP server, destined to the CMTS, as anything other than 67/67.

> We had to downgrade the CMTS to get our subscribers back to a good state,
> but I've been able to simulate the problem by using bittcap to inject
> earlier-captured DHCP Discover packets into the network.  When I change
the
> source port to 68/67 using bittcape, those DHCP Discovers result in proper
> DHCP Offers of 67/67.

I think if the variable you changed is the CMTS, that it is hard to
imagine the bug being elsewhere.

> Has anyone else seen this, does newer ISC dhcpd code resolve this, and if
> not, where in the code is this being done so that this can be fixed?

This is news to me.  I don't think the dhcp server should care what
source port the relay uses, so long as it uses destination port 67.

-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		you'll just have to do it again."
Internet Systems Consortium, Inc.	-- Jack T. Hankins