Strange Issue with Linksys routers

Ben Wiechman benw at meltel.com
Mon May 18 16:37:33 UTC 2009


We were getting 4 at the server because the relay was getting 4 from the
network. The customer router would send out a request and that request would
be mirrored by the Belkin routers in question - packet appeared the same
with the exception of the source mac address. How a device mirrors a unicast
request, even on the same broadcast domain, is a bit strange, but that is
what happened. Cut off service for the subscribers with the apparently
malicious Belkin routers and the duplicate requests ended. We have three of
those routers in our lab now so we'll see if we can replicate that issue. 

Strangely they didn't seem to duplicate every request, so if they only
duplicated requests with a matching transaction ID or something else remains
to be seen.


> -----Original Message-----
> From: dhcp-users-bounces at lists.isc.org [mailto:dhcp-users-
> bounces at lists.isc.org] On Behalf Of Frank Bulk
> Sent: Monday, May 18, 2009 9:54 AM
> To: Ben Wiechman; Users of ISC DHCP
> Subject: RE: Strange Issue with Linksys routers
> 
> You should see what your DHCP relays are doing with those packets....you
> shouldn't be getting 4.
> 
> Frank
> 
> -----Original Message-----
> From: dhcp-users-bounces at lists.isc.org
> [mailto:dhcp-users-bounces at lists.isc.org] On Behalf Of Ben Wiechman
> Sent: Monday, May 18, 2009 9:00 AM
> To: Users of ISC DHCP
> Subject: RE: Strange Issue with Linksys routers
> 
> I must have missed those discussions.
> 
> It was a long week. But speaking of Belkins we found a couple of those in
> the network that appeared to be duplicating unicast DHCP requests.
Strangest
> thing. We were trying to get to the bottom of this issue and doing
captures
> at an affected customers location and in the data center. We'd see a
single
> unicast request at the customer location, but 4 in the data center and the
> customer obviously got four responses. Doing a packet capture on the
network
> segment showed a single request with the source mac of the customer's
> Linksys router, and 4 duplicate requests from different Belkin routers
with
> identical information with the exception of the source mac address in the
IP
> headers. Goofy.
> 
> Ben Wiechman
> Network Administrator
> 
> 
> > -----Original Message-----
> > From: Frank Bulk [mailto:frnkblk at iname.com]
> > Sent: Sunday, May 17, 2009 8:49 PM
> > To: Ben Wiechman; Users of ISC DHCP
> > Subject: RE: Strange Issue with Linksys routers
> >
> > Good point -- we've seen brownouts do the same.  Almost always get a
rash
> of
> > helpdesk calls after those.  I would prefer the power company just do a
> > 5-second salute. =)
> >
> > It has been previously discussed, but it's worth mentioning again the
> Belkin
> > DHCP issue.  We dealt with most of them a few months ago but still see
> them
> > pop up once in a while.  Again, a firmware upgrade fixes most of those.
> >
> > Sometimes I joke that the SOHO vendors ought to be paying us to do their
> > tech support.
> >
> > Frank
> >
> > -----Original Message-----
> > From: dhcp-users-bounces at lists.isc.org
> > [mailto:dhcp-users-bounces at lists.isc.org] On Behalf Of Ben Wiechman
> > Sent: Sunday, May 17, 2009 8:42 PM
> > To: Users of ISC DHCP
> > Subject: RE: Strange Issue with Linksys routers
> >
> > We've seen similar issues in the past after lightning storms where the
> > routers seem to be affected by a brownout or other power issue.
Typically
> a
> > reboot seems to cure those issues. In this case these routers are
> scattered
> > across several separate power companies and 50-60 miles of geography.
> > Rebooting the router does not clear up the issue. Upgrading the firmware
> > seems to be the only real solution that has a long term affect.
> >
> > Ben Wiechman
> > Network Administrator
> >
> > > -----Original Message-----
> > > From: dhcp-users-bounces at lists.isc.org [mailto:dhcp-users-
> > > bounces at lists.isc.org] On Behalf Of Frank Bulk
> > > Sent: Sunday, May 17, 2009 8:17 PM
> > > To: Users of ISC DHCP
> > > Subject: RE: Strange Issue with Linksys routers
> > >
> > > Ben:
> > >
> > > We have hundreds of WRT54G's attached to cable modems and not noticed
> this
> > > issue.  Any chance that there was a common powering/lightning issue
that
> > > affected these routers' power supplies?  We have seen several cases
> where
> > a
> > > Linksys router acts marginally with a bad power supply.
> > >
> > > Frank
> > >
> > > -----Original Message-----
> > > From: dhcp-users-bounces at lists.isc.org
> > > [mailto:dhcp-users-bounces at lists.isc.org] On Behalf Of Ben Wiechman
> > > Sent: Saturday, May 16, 2009 5:21 PM
> > > To: Users of ISC DHCP
> > > Subject: Strange Issue with Linksys routers
> > >
> > > This doesn't appear to be an issue that is specific to ISC, but since
> this
> > > is most readily visible in the dhcp logs maybe someone else here has
> seen
> > or
> > > is seeing this issue as well.
> > >
> > > We have a number of subscribers on our network (75-100 - possibly
more)
> > that
> > > are running WRT54G/GS routers with older firmware (1.00.6/7, etc.)
with
> > > ethernet controllers that appear to lock up. When the router is power
> > cycled
> > > they tend to work properly for several hours and then the ethernet
> > > controller appears to lock up again. This keeps repeating.
> > >
> > > When the condition exists any computers attached to the wired
interfaces
> > > will eventually report limited or no connectivity. Any traffic sent to
> the
> > > router on the LAN ports receives no response. However it is possible
to
> > > connect to the routers using the wireless interface and access the web
> > > management interface. With remote management enabled and pings allowed
> > they
> > > will not respond on the WAN interface when this condition exists. We
> have
> > > done packet captures to verify that the icmp/ip packets are being
> > delivered
> > > to the WAN interface, however the router generates no response.
> > >
> > > This is where it gets weird. The router will continue to send
> > > DHCPREQUEST/DHCPOFFER packets but does not appear to receive the
> response.
> > > This is how we initially noticed the issue. Large numbers of routers
> were
> > > hammering our dhcp server hundreds of times every hour with
DHCPDISCOVER
> > > broadcasts.
> > >
> > > When the router is power cycled it will broadcast a DHCPDISCOVER
packet
> to
> > > the dhcp server, receive the offer, broadcast the request and receive
> the
> > > ack. Our default lease time is 12 hours. Normally the router would
send
> a
> > > unicast DHCPREQUEST to the server half way through the lease time and
> > > receives a unicast DHCPACK. Under normal conditions this would simply
> > > repeat. Here the ethernet controller appears to lock up at some point.
> So
> > > the router will send the DHCPREQUEST packet at the midway point, then
> with
> > > increasing frequency as the end of the lease period nears. Doing a
> packet
> > > capture will show the DHCPACK is received at the WAN interface of the
> > > router. In the last couple of minutes before the lease expires the
> router
> > > will broadcast a series of DHCPREQUEST packets and receive broadcast
> > > responses from the server. Once the dhcp lease expires the router will
> > > continue to broadcast a series of DHCPDISCOVER messages every minute
or
> so
> > > and receives the DHCPOFFERs in return. This will repeat until the
router
> > is
> > > power cycled. Even if the ethernet ports are disconnected the lost
link
> is
> > > not detected.
> > >
> > > The dicover/offer cycle can be tripped by logging into the router via
> the
> > > wireless interface and changing the hostname. This causes the router
to
> > send
> > > a DHCPRELEASE request, followed by a DHCPDISCOVER. It receives a
> > DHCPOFFER,
> > > however does not appear to process the offer and once again enters a
> loop
> > > where it continues to broadcast a series of DHCPDISCOVER packets every
> > > minute or so.
> > >
> > > We have not seen this on WRT54GL routers to this point. It appears to
> have
> > > begun at a very defined point on Monday 5/11. Is this some new
exploit?
> We
> > > have yet to track down anything that appears to trigger this
condition.
> > >
> > > Has anyone seen anything like this in the past?
> > >
> > > Ben Wiechman
> > > Network Administrator
> > > Wisper High Speed Internet
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > dhcp-users mailing list
> > > dhcp-users at lists.isc.org
> > > https://lists.isc.org/mailman/listinfo/dhcp-users
> > >
> > > _______________________________________________
> > > dhcp-users mailing list
> > > dhcp-users at lists.isc.org
> > > https://lists.isc.org/mailman/listinfo/dhcp-users
> > >
> >
> >
> >
> > _______________________________________________
> > dhcp-users mailing list
> > dhcp-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/dhcp-users
> >
> 
> 
> 
> _______________________________________________
> dhcp-users mailing list
> dhcp-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/dhcp-users
> 
> _______________________________________________
> dhcp-users mailing list
> dhcp-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/dhcp-users
> 






More information about the dhcp-users mailing list