RPZ and www.rackspace.com
John Miller
johnmill at brandeis.edu
Fri May 30 17:42:33 UTC 2014
It's surprising that more organizations don't fix this--it can be a
serious DoS vulnerability if the record is important enough. Anyone
know of tools that, given a zone or a set of labels, will test for this
behavior?
John
On 05/30/2014 11:42 AM, David A. Evans wrote:
> To my question of how many more are lurking out there. It
> looks like quite a few. I am not sure we are going to be able to
> continue with RPZ's and NSDNAME's.
>
> xserv.dell.com is my newest main stream web site having the
> issue.
>
> I is behaving the same way as www.rackspace.com. They have
> DNS servers answering NXDOMAIN when they should be answering NODATA.
>
>
>
>
>
>
> *David A. Evans*
> *Enterprise IP/DNS Management*
> *Network Infrastructure Tools and Services*
> *(309) 675-9700 *
> *_Evans_David_A at cat.com_* <mailto:Evans_David_A at cat.com>
>
> bind-users-bounces at lists.isc.org wrote on 05/28/2014 03:09:54 PM:
>
> > From: "David A. Evans" <Evans_David_A at cat.com>
> > To: bind-users at isc.org
> > Date: 05/28/2014 03:10 PM
> > Subject: Re: RPZ and www.rackspace.com
> > Sent by: bind-users-bounces at lists.isc.org
> >
> > CATERPILLAR SECURITY ALERT: The email address in the sender line
> > does not match the account that sent the email. This can be an
> > indication of phishing. Do not click links or open attachments
> > unless you are certain it is from a safe source. Learn more at
> > security.cat.com/phishing
> > Rack Space appears to have fixed the issue. "dig
> > www.wip.rackspace.com NS" now returns NO DATA instead of NXDOMAIN.
> >
> > I wonder how many more are lurking out there.
> >
> >
> > We are still getting a trickle in of complaints about
> > slowness and failures that appear to be related to the RPZ and the
> > amount of time it takes to complete all the extra queries for the
> > NSDNAME checks. When we research these they seem to fit into 2 groups.
> >
> >
> > 1. DNS zones with "slightly" broken infrastructure. These
> > would be domains with either slow response from one or more name
> > servers or not responding name servers. A recursive resolver
> > without RPZ loaded can work though the issue but the extra lookups
> > required primarily for the NSDNAME increases the query time to the
> > point where DNS times out from the client perspective. I can't
> > really see a fix here, the issue does reside with the domain owner,
> > we are simply more susceptible to the issue because of the RPZ's.
> >
> > 2. DNS zones with a large number of NS records and the name
> > servers have FQDN's in several different DNS zones.
> >
> > www.domain.com NS ns1.domain1.com
> > NS ns2.domain2.com
> > NS ns3.domain3.net .....
> >
> > These are the most frustrating as there is really nothing
> > wrong with this setup in my opinion. This is just going to
> > generate a huge number of DNS lookups to do a full NSDNAME check.
> > These are hard to explain away as they "work from home" and "work
> > from my phone".
> >
> >
> > Have others found similar issues when implementing RPZ's?
> >
> >
> >
> >
> >
> >
> > David A. Evans
> > Enterprise IP/DNS Management
> > Network Infrastructure Tools and Services
> >
> >
> >
> > Mark Andrews <marka at isc.org> wrote on 05/07/2014 10:44:05 AM:
> >
> > > From: Mark Andrews <marka at isc.org>
> > > To: "David A. Evans" <Evans_David_A at cat.com>
> > > Cc: bind-users at isc.org
> > > Date: 05/07/2014 10:44 AM
> > > Subject: Re: RPZ and www.rackspace.com
> > >
> > >
> > > In message <OFDC3C86D9.D668B707-ON86257CD1.005339FC-86257CD1.
> > > 005431EB at notes.cat.com>, "David A. Evans" writes:
> > > >
> > > > I've done some more troubleshooting with info from people
> that
> > > > responded directly to me and not to the list. This can be
> reproduced
> > > > without any RPZ loaded by mimicking the behavior of the RPZ lookups
> > > > required to validate NSDNAME lines.
> > > >
> > > > Issue these 'digs' within 30 second of each other.
> > > >
> > > > dig www.wip.rackspace.com
> > > > www.wip.rackspace.com. 30 IN A 173.203.44.116
> > > >
> > > > dig www.wip.rackspace.com NS
> > > > (NXDOMAIN)
> > > >
> > > > dig www.wip.rackspace.com
> > > > (NXDOMAIN)
> > > >
> > > >
> > > > I think this is another case of miss configured load
> balancers.
> > > > Shouldn't the NS record lookup respond with a NODATA response and
> not
> > > > NXDOMAIN?
> > >
> > > Yes. The name exists.
> > >
> > > > That still doesn't really answer why a site as big as
> > > > www.rackspace.comisn't getting hit with support issues on
> theirweb site.
> > > > It only took us about 4 hours into our first production day with
> > > > NSDNAME's in our RPZ to get a call about www.rackspace.comnot
> loading.
> > >
> > > Because NS queries are not common with normal DNS lookups. For
> > > some reason people that deploy load balancers think they don't need
> > > to fix issues like this. Send something other than a A record and
> > > you get:
> > >
> > > - NXDOMAIN being returned when the name exists.
> > > - NOTIMP being returned.
> > > (Really you can't just send NODATA?)
> > > - REFUSED being returned.
> > > (Really you don't want to tell us the record does not exist?)
> > > - the wrong SOA being returned.
> > > - malformed RDATA with the content being the A record content.
> > >
> > > Mark
> > >
> > > > David A. Evans
> > > > Enterprise IP/DNS Management
> > > > Network Infrastructure Tools and Services
> > > > Evans_David_A at cat.com
> > > >
> > > >
> > > >
> > > > From: "David A. Evans" <Evans_David_A at cat.com>
> > > > To: bind-users at lists.isc.org
> > > > Date: 05/07/2014 09:11 AM
> > > > Subject: RPZ and www.rackspace.com
> > > > Sent by: bind-users-bounces at lists.isc.org
> > > >
> > > >
> > > >
> > > > CATERPILLAR SECURITY ALERT: The email address in the sender
> linedoes not
> > > > match the account that sent the email. This can be an indication of
> > > > phishing. Do not click links or open attachments unless you are
> > certain it
> > > > is from a safe source. Learn more at security.cat.com/phishing
> > > > We have just enabled RPZ with some NSDNAME checks and are
> seeing
> > > > an issue resolving www.rackspace.com.
> > > >
> > > > The first lookup is successful and returns both the CNAME
> and the
> > > > A record. The second query, within a second of the first, will only
> > > > return the CNAME. It will only return the CNAME until the TTL of
> the A
> > > > record times out. The first query, when it actually has to go
> out and do
> > > > recursion will always work. Answering from cache will always
> fail. When
> > > > you inspect the cache during the time that it is only returning
> > the CNAME,
> > > > the record in cache is "www.wip.rackspace.com type ANY
> > NXDOMAIN". This
> > > > only happens with RPZ's enabled and query hitting a RPZ zone with a
> > > > NSDNAME line. Turning off RPZ or whitelisting the lookup via
> RPZ before
> > > > it hits a RPZ with NSDNAME allows the query to be successful 100%
> of the
> > > > time.
> > > >
> > > >
> > > > Can anyone else verify this behavior? What is going on
> with
> > > > www.rackspace.com? If this is a miss configuration on
> Rackspace's DNS
> > > > servers how are they not getting hit with support calls like crazy?
> > > >
> > > >
> > > >
> > > > dig @redacted.cat.com www.rackspace.com
> > > >
> > > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> @redacted.cat.com
> > > > www.rackspace.com
> > > > ; (1 server found)
> > > > ;; global options: +cmd
> > > > ;; Got answer:
> > > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337
> > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> > > >
> > > > ;; QUESTION SECTION:
> > > > ;www.rackspace.com. IN A
> > > >
> > > > ;; ANSWER SECTION:
> > > > www.rackspace.com. 300 IN CNAME www.wip.rackspace.com.
> > > > www.wip.rackspace.com. 30 IN A 173.203.44.116
> > > >
> > > > ;; Query time: 193 msec
> > > > ;; SERVER: redacted
> > > > ;; WHEN: Wed May 7 08:53:08 2014
> > > > ;; MSG SIZE rcvd: 73
> > > >
> > > >
> > > >
> > > > dig @redacted.cat.com www.rackspace.com
> > > >
> > > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> @redacted.cat.com
> > > > www.rackspace.com
> > > > ; (1 server found)
> > > > ;; global options: +cmd
> > > > ;; Got answer:
> > > > ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905
> > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
> > > >
> > > > ;; QUESTION SECTION:
> > > > ;www.rackspace.com. IN A
> > > >
> > > > ;; ANSWER SECTION:
> > > > www.rackspace.com. 298 IN CNAME www.wip.rackspace.com.
> > > >
> > > > ;; AUTHORITY SECTION:
> > > > wip.rackspace.com. 58 IN SOA www-gtm-ord1.rackspace.com.
> > > > hostmaster.305181-GTM1.rackspace.com. 86 10800 3600 604800 60
> > > >
> > > > ;; Query time: 2 msec
> > > > ;; SERVER: redacted
> > > > ;; WHEN: Wed May 7 08:53:10 2014
> > > > ;; MSG SIZE rcvd: 129
> > > >
> > > >
> > > > David A. Evans
> > > > Enterprise IP/DNS Management
> > > > Network Infrastructure Tools and Services
> > > > Evans_David_A at cat.com _______________________________________________
> > > > Please visit https://lists.isc.org/mailman/listinfo/bind-usersto
> > > > unsubscribe from this list
> > > >
> > > > bind-users mailing list
> > > > bind-users at lists.isc.org
> > > > https://lists.isc.org/mailman/listinfo/bind-users
> > > >
> > > > --=_alternative 0054318186257CD1_=
> > > > Content-Type: text/html; charset="US-ASCII"
> > > > _______________________________________________
> > Please visit https://lists.isc.org/mailman/listinfo/bind-usersto
> > unsubscribe from this list
> >
> > bind-users mailing list
> > bind-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/bind-users
>
>
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
More information about the bind-users
mailing list