RPZ and www.rackspace.com
Mark Andrews
marka at isc.org
Fri May 30 23:48:39 UTC 2014
They don't fix it because they are not usually aware of it. The
vast bulk of the queries against load balancers are A and AAAA
queries and long as these don't return NXDOMAIN they don't see
operational problems. Servfail for AAAA is not a issue for them
as it doesn't impact the A lookup which is all they care about.
NXDOMAIN does impact on A lookups.
Now there are a number of different errors and fixing each of these
requires different solutions.
Incorrect NXDOMAIN.
xserv.ins.dell.com returns NOERROR for A and AAAA queries which
make up the bulk of the queries the name gets. NXDOMAIN is wrong
and there is a CERT advisory for NXDOMAIN against AAAA queries.
We could go to CERT and get them to issue / re-issue the advisory
mentioning NS records explicitly.
I write to the operators when I see a error requesting that they
contact their nameserver vendor for a fix. The message contains a
description of the error and how to reproduce it. It also contains
a description of what the correct response is. This sometimes I
get a acknowledgement. Sometimes the operator opens a ticket with
the vendor.
Error fall into a number of classes.
* no response (nameserver vendor needs to fix or firewall needs to be adjusted)
* type specific wrong rcode (NOTIMP, REFUSED, NXDOMAIN - correct SOA) requires
the nameserver vendor to fix. Request they contact the nameserver vendor
for a fix.
* wrong SOA (NXDOMAIN or NOERROR) is usually a configuration issue the wrong
backing zone has been configured. Request they contact the nameserver
vendor for correct configuration instructions.
If talking to the privately doesn't work then "name and shame" is
the next step.
Mark
In message <5388C309.5000700 at brandeis.edu>, John Miller writes:
> It's surprising that more organizations don't fix this--it can be a
> serious DoS vulnerability if the record is important enough. Anyone
> know of tools that, given a zone or a set of labels, will test for this
> behavior?
>
> John
>
> On 05/30/2014 11:42 AM, David A. Evans wrote:
> > To my question of how many more are lurking out there. It
> > looks like quite a few. I am not sure we are going to be able to
> > continue with RPZ's and NSDNAME's.
> >
> > xserv.dell.com is my newest main stream web site having the
> > issue.
> >
> > I is behaving the same way as www.rackspace.com. They have
> > DNS servers answering NXDOMAIN when they should be answering NODATA.
> >
> >
> >
> >
> >
> >
> > *David A. Evans*
> > *Enterprise IP/DNS Management*
> > *Network Infrastructure Tools and Services*
> > *(309) 675-9700 *
> > *_Evans_David_A at cat.com_* <mailto:Evans_David_A at cat.com>
> >
> > bind-users-bounces at lists.isc.org wrote on 05/28/2014 03:09:54 PM:
> >
> > > From: "David A. Evans" <Evans_David_A at cat.com>
> > > To: bind-users at isc.org
> > > Date: 05/28/2014 03:10 PM
> > > Subject: Re: RPZ and www.rackspace.com
> > > Sent by: bind-users-bounces at lists.isc.org
> > >
> > > CATERPILLAR SECURITY ALERT: The email address in the sender line
> > > does not match the account that sent the email. This can be an
> > > indication of phishing. Do not click links or open attachments
> > > unless you are certain it is from a safe source. Learn more at
> > > security.cat.com/phishing
> > > Rack Space appears to have fixed the issue. "dig
> > > www.wip.rackspace.com NS" now returns NO DATA instead of NXDOMAIN.
> > >
> > > I wonder how many more are lurking out there.
> > >
> > >
> > > We are still getting a trickle in of complaints about
> > > slowness and failures that appear to be related to the RPZ and the
> > > amount of time it takes to complete all the extra queries for the
> > > NSDNAME checks. When we research these they seem to fit into 2 groups.
> > >
> > >
> > > 1. DNS zones with "slightly" broken infrastructure. These
> > > would be domains with either slow response from one or more name
> > > servers or not responding name servers. A recursive resolver
> > > without RPZ loaded can work though the issue but the extra lookups
> > > required primarily for the NSDNAME increases the query time to the
> > > point where DNS times out from the client perspective. I can't
> > > really see a fix here, the issue does reside with the domain owner,
> > > we are simply more susceptible to the issue because of the RPZ's.
> > >
> > > 2. DNS zones with a large number of NS records and the name
> > > servers have FQDN's in several different DNS zones.
> > >
> > > www.domain.com NS ns1.domain1.com
> > > NS ns2.domain2.com
> > > NS ns3.domain3.net .....
> > >
> > > These are the most frustrating as there is really nothing
> > > wrong with this setup in my opinion. This is just going to
> > > generate a huge number of DNS lookups to do a full NSDNAME check.
> > > These are hard to explain away as they "work from home" and "work
> > > from my phone".
> > >
> > >
> > > Have others found similar issues when implementing RPZ's?
> > >
> > >
> > >
> > >
> > >
> > >
> > > David A. Evans
> > > Enterprise IP/DNS Management
> > > Network Infrastructure Tools and Services
> > >
> > >
> > >
> > > Mark Andrews <marka at isc.org> wrote on 05/07/2014 10:44:05 AM:
> > >
> > > > From: Mark Andrews <marka at isc.org>
> > > > To: "David A. Evans" <Evans_David_A at cat.com>
> > > > Cc: bind-users at isc.org
> > > > Date: 05/07/2014 10:44 AM
> > > > Subject: Re: RPZ and www.rackspace.com
> > > >
> > > >
> > > > In message <OFDC3C86D9.D668B707-ON86257CD1.005339FC-86257CD1.
> > > > 005431EB at notes.cat.com>, "David A. Evans" writes:
> > > > >
> > > > > I've done some more troubleshooting with info from people
> > that
> > > > > responded directly to me and not to the list. This can be
> > reproduced
> > > > > without any RPZ loaded by mimicking the behavior of the RPZ lookups
> > > > > required to validate NSDNAME lines.
> > > > >
> > > > > Issue these 'digs' within 30 second of each other.
> > > > >
> > > > > dig www.wip.rackspace.com
> > > > > www.wip.rackspace.com. 30 IN A 173.203.44.116
> > > > >
> > > > > dig www.wip.rackspace.com NS
> > > > > (NXDOMAIN)
> > > > >
> > > > > dig www.wip.rackspace.com
> > > > > (NXDOMAIN)
> > > > >
> > > > >
> > > > > I think this is another case of miss configured load
> > balancers.
> > > > > Shouldn't the NS record lookup respond with a NODATA response and
> > not
> > > > > NXDOMAIN?
> > > >
> > > > Yes. The name exists.
> > > >
> > > > > That still doesn't really answer why a site as big as
> > > > > www.rackspace.comisn't getting hit with support issues on
> > theirweb site.
> > > > > It only took us about 4 hours into our first production day with
> > > > > NSDNAME's in our RPZ to get a call about www.rackspace.comnot
> > loading.
> > > >
> > > > Because NS queries are not common with normal DNS lookups. For
> > > > some reason people that deploy load balancers think they don't need
> > > > to fix issues like this. Send something other than a A record and
> > > > you get:
> > > >
> > > > - NXDOMAIN being returned when the name exists.
> > > > - NOTIMP being returned.
> > > > (Really you can't just send NODATA?)
> > > > - REFUSED being returned.
> > > > (Really you don't want to tell us the record does not exist?)
> > > > - the wrong SOA being returned.
> > > > - malformed RDATA with the content being the A record content.
> > > >
> > > > Mark
> > > >
> > > > > David A. Evans
> > > > > Enterprise IP/DNS Management
> > > > > Network Infrastructure Tools and Services
> > > > > Evans_David_A at cat.com
> > > > >
> > > > >
> > > > >
> > > > > From: "David A. Evans" <Evans_David_A at cat.com>
> > > > > To: bind-users at lists.isc.org
> > > > > Date: 05/07/2014 09:11 AM
> > > > > Subject: RPZ and www.rackspace.com
> > > > > Sent by: bind-users-bounces at lists.isc.org
> > > > >
> > > > >
> > > > >
> > > > > CATERPILLAR SECURITY ALERT: The email address in the sender
> > linedoes not
> > > > > match the account that sent the email. This can be an indication of
> > > > > phishing. Do not click links or open attachments unless you are
> > > certain it
> > > > > is from a safe source. Learn more at security.cat.com/phishing
> > > > > We have just enabled RPZ with some NSDNAME checks and are
> > seeing
> > > > > an issue resolving www.rackspace.com.
> > > > >
> > > > > The first lookup is successful and returns both the CNAME
> > and the
> > > > > A record. The second query, within a second of the first, will only
> > > > > return the CNAME. It will only return the CNAME until the TTL of
> > the A
> > > > > record times out. The first query, when it actually has to go
> > out and do
> > > > > recursion will always work. Answering from cache will always
> > fail. When
> > > > > you inspect the cache during the time that it is only returning
> > > the CNAME,
> > > > > the record in cache is "www.wip.rackspace.com type ANY
> > > NXDOMAIN". This
> > > > > only happens with RPZ's enabled and query hitting a RPZ zone with a
> > > > > NSDNAME line. Turning off RPZ or whitelisting the lookup via
> > RPZ before
> > > > > it hits a RPZ with NSDNAME allows the query to be successful 100%
> > of the
> > > > > time.
> > > > >
> > > > >
> > > > > Can anyone else verify this behavior? What is going on
> > with
> > > > > www.rackspace.com? If this is a miss configuration on
> > Rackspace's DNS
> > > > > servers how are they not getting hit with support calls like crazy?
> > > > >
> > > > >
> > > > >
> > > > > dig @redacted.cat.com www.rackspace.com
> > > > >
> > > > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> > @redacted.cat.com
> > > > > www.rackspace.com
> > > > > ; (1 server found)
> > > > > ;; global options: +cmd
> > > > > ;; Got answer:
> > > > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337
> > > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> > > > >
> > > > > ;; QUESTION SECTION:
> > > > > ;www.rackspace.com. IN A
> > > > >
> > > > > ;; ANSWER SECTION:
> > > > > www.rackspace.com. 300 IN CNAME www.wip.rackspace.com.
> > > > > www.wip.rackspace.com. 30 IN A 173.203.44.116
> > > > >
> > > > > ;; Query time: 193 msec
> > > > > ;; SERVER: redacted
> > > > > ;; WHEN: Wed May 7 08:53:08 2014
> > > > > ;; MSG SIZE rcvd: 73
> > > > >
> > > > >
> > > > >
> > > > > dig @redacted.cat.com www.rackspace.com
> > > > >
> > > > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> > @redacted.cat.com
> > > > > www.rackspace.com
> > > > > ; (1 server found)
> > > > > ;; global options: +cmd
> > > > > ;; Got answer:
> > > > > ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905
> > > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
> > > > >
> > > > > ;; QUESTION SECTION:
> > > > > ;www.rackspace.com. IN A
> > > > >
> > > > > ;; ANSWER SECTION:
> > > > > www.rackspace.com. 298 IN CNAME www.wip.rackspace.com.
> > > > >
> > > > > ;; AUTHORITY SECTION:
> > > > > wip.rackspace.com. 58 IN SOA www-gtm-ord1.rackspace.com.
> > > > > hostmaster.305181-GTM1.rackspace.com. 86 10800 3600 604800 60
> > > > >
> > > > > ;; Query time: 2 msec
> > > > > ;; SERVER: redacted
> > > > > ;; WHEN: Wed May 7 08:53:10 2014
> > > > > ;; MSG SIZE rcvd: 129
> > > > >
> > > > >
> > > > > David A. Evans
> > > > > Enterprise IP/DNS Management
> > > > > Network Infrastructure Tools and Services
> > > > > Evans_David_A at cat.com _______________________________________________
> > > > > Please visit https://lists.isc.org/mailman/listinfo/bind-usersto
> > > > > unsubscribe from this list
> > > > >
> > > > > bind-users mailing list
> > > > > bind-users at lists.isc.org
> > > > > https://lists.isc.org/mailman/listinfo/bind-users
> > > > >
> > > > > --=_alternative 0054318186257CD1_=
> > > > > Content-Type: text/html; charset="US-ASCII"
> > > > > _______________________________________________
> > > Please visit https://lists.isc.org/mailman/listinfo/bind-usersto
> > > unsubscribe from this list
> > >
> > > bind-users mailing list
> > > bind-users at lists.isc.org
> > > https://lists.isc.org/mailman/listinfo/bind-users
> >
> >
> > _______________________________________________
> > Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
> >
> > bind-users mailing list
> > bind-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/bind-users
> >
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka at isc.org
More information about the bind-users
mailing list