RPZ and www.rackspace.com
Mark Andrews
marka at isc.org
Wed May 7 15:44:05 UTC 2014
In message <OFDC3C86D9.D668B707-ON86257CD1.005339FC-86257CD1.005431EB at notes.cat.com>, "David A. Evans" writes:
>
> I've done some more troubleshooting with info from people that
> responded directly to me and not to the list. This can be reproduced
> without any RPZ loaded by mimicking the behavior of the RPZ lookups
> required to validate NSDNAME lines.
>
> Issue these 'digs' within 30 second of each other.
>
> dig www.wip.rackspace.com
> www.wip.rackspace.com. 30 IN A 173.203.44.116
>
> dig www.wip.rackspace.com NS
> (NXDOMAIN)
>
> dig www.wip.rackspace.com
> (NXDOMAIN)
>
>
> I think this is another case of miss configured load balancers.
> Shouldn't the NS record lookup respond with a NODATA response and not
> NXDOMAIN?
Yes. The name exists.
> That still doesn't really answer why a site as big as
> www.rackspace.com isn't getting hit with support issues on their web site.
> It only took us about 4 hours into our first production day with
> NSDNAME's in our RPZ to get a call about www.rackspace.com not loading.
Because NS queries are not common with normal DNS lookups. For
some reason people that deploy load balancers think they don't need
to fix issues like this. Send something other than a A record and
you get:
- NXDOMAIN being returned when the name exists.
- NOTIMP being returned.
(Really you can't just send NODATA?)
- REFUSED being returned.
(Really you don't want to tell us the record does not exist?)
- the wrong SOA being returned.
- malformed RDATA with the content being the A record content.
Mark
> David A. Evans
> Enterprise IP/DNS Management
> Network Infrastructure Tools and Services
> Evans_David_A at cat.com
>
>
>
> From: "David A. Evans" <Evans_David_A at cat.com>
> To: bind-users at lists.isc.org
> Date: 05/07/2014 09:11 AM
> Subject: RPZ and www.rackspace.com
> Sent by: bind-users-bounces at lists.isc.org
>
>
>
> CATERPILLAR SECURITY ALERT: The email address in the sender line does not
> match the account that sent the email. This can be an indication of
> phishing. Do not click links or open attachments unless you are certain it
> is from a safe source. Learn more at security.cat.com/phishing
> We have just enabled RPZ with some NSDNAME checks and are seeing
> an issue resolving www.rackspace.com.
>
> The first lookup is successful and returns both the CNAME and the
> A record. The second query, within a second of the first, will only
> return the CNAME. It will only return the CNAME until the TTL of the A
> record times out. The first query, when it actually has to go out and do
> recursion will always work. Answering from cache will always fail. When
> you inspect the cache during the time that it is only returning the CNAME,
> the record in cache is "www.wip.rackspace.com type ANY NXDOMAIN". This
> only happens with RPZ's enabled and query hitting a RPZ zone with a
> NSDNAME line. Turning off RPZ or whitelisting the lookup via RPZ before
> it hits a RPZ with NSDNAME allows the query to be successful 100% of the
> time.
>
>
> Can anyone else verify this behavior? What is going on with
> www.rackspace.com? If this is a miss configuration on Rackspace's DNS
> servers how are they not getting hit with support calls like crazy?
>
>
>
> dig @redacted.cat.com www.rackspace.com
>
> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> @redacted.cat.com
> www.rackspace.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;www.rackspace.com. IN A
>
> ;; ANSWER SECTION:
> www.rackspace.com. 300 IN CNAME www.wip.rackspace.com.
> www.wip.rackspace.com. 30 IN A 173.203.44.116
>
> ;; Query time: 193 msec
> ;; SERVER: redacted
> ;; WHEN: Wed May 7 08:53:08 2014
> ;; MSG SIZE rcvd: 73
>
>
>
> dig @redacted.cat.com www.rackspace.com
>
> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> @redacted.cat.com
> www.rackspace.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;www.rackspace.com. IN A
>
> ;; ANSWER SECTION:
> www.rackspace.com. 298 IN CNAME www.wip.rackspace.com.
>
> ;; AUTHORITY SECTION:
> wip.rackspace.com. 58 IN SOA www-gtm-ord1.rackspace.com.
> hostmaster.305181-GTM1.rackspace.com. 86 10800 3600 604800 60
>
> ;; Query time: 2 msec
> ;; SERVER: redacted
> ;; WHEN: Wed May 7 08:53:10 2014
> ;; MSG SIZE rcvd: 129
>
>
> David A. Evans
> Enterprise IP/DNS Management
> Network Infrastructure Tools and Services
> Evans_David_A at cat.com _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
> --=_alternative 0054318186257CD1_=
> Content-Type: text/html; charset="US-ASCII"
>
> <font size=2 face="sans-serif"> I've
> done some more troubleshooting with info from people that responded directly
> to me and not to the list. This can be reproduced without
> any RPZ loaded by mimicking the behavior of the RPZ lookups required to
> validate NSDNAME lines.</font>
> <br>
> <br><font size=2 face="sans-serif">Issue these 'digs' within 30 second
> of each other.</font>
> <br>
> <br><font size=2 face="sans-serif">dig </font><a href=www.wip.rackspace.com><font size=2 face="sans-serif">www.wip.rac
> kspace.com</font></a><font size=2 face="sans-serif">
> </font>
> <br><a href=www.wip.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.wip.rackspace.com</u></font></a><fo
> nt size=2 face="sans-serif">.
> 30 IN A
> 173.203.44.116</font><font size=3> </font>
> <br><font size=3><br>
> </font><font size=2 face="sans-serif">dig </font><a href=www.wip.rackspace.com><font size=2 face="sans-serif">www.wip.
> rackspace.com</font></a><font size=2 face="sans-serif">
> NS</font>
> <br><font size=2 face="sans-serif">(NXDOMAIN)</font>
> <br>
> <br><font size=2 face="sans-serif">dig </font><a href=www.wip.rackspace.com><font size=2 face="sans-serif">www.wip.rac
> kspace.com</font></a>
> <br><font size=2 face="sans-serif">(NXDOMAIN)</font>
> <br>
> <br>
> <br><font size=2 face="sans-serif"> I
> think this is another case of miss configured load balancers. Shouldn't
> the NS record lookup respond with a NODATA response and not NXDOMAIN? </font>
> <br>
> <br><font size=2 face="sans-serif"> That
> still doesn't really answer why a site as big as </font><a href=www.rackspace.com><font size=2 face="sans-serif">www.r
> ackspace.com</font></a><font size=2 face="sans-serif">
> isn't getting hit with support issues on their web site. It only
> took us about 4 hours into our first production day with NSDNAME's in our
> RPZ to get a call about </font><a href=www.rackspace.com><font size=2 face="sans-serif">www.rackspace.com</font></a><f
> ont size=2 face="sans-serif">
> not loading.</font>
> <br>
> <br>
> <br>
> <br><font size=5 color=blue><b>David A. Evans</b></font>
> <br><font size=3><b>Enterprise IP/DNS Management</b></font>
> <br><font size=3><b>Network Infrastructure Tools and Services</b></font>
> <br><a href=mailto:Evans_David_A at cat.com><font size=3 color=blue><b><u>Evans_David_A at cat.com</u></b></font></a>
> <br>
> <br>
> <br>
> <br><font size=1 color=#5f5f5f face="sans-serif">From:
> </font><font size=1 face="sans-serif">"David A. Evans"
> <Evans_David_A at cat.com></font>
> <br><font size=1 color=#5f5f5f face="sans-serif">To:
> </font><font size=1 face="sans-serif">bind-users at lists.isc.org</font>
> <br><font size=1 color=#5f5f5f face="sans-serif">Date:
> </font><font size=1 face="sans-serif">05/07/2014 09:11 AM</font>
> <br><font size=1 color=#5f5f5f face="sans-serif">Subject:
> </font><font size=1 face="sans-serif">RPZ and </font><a href=www.rackspace.com><font size=1 face="sans-se
> rif">www.rackspace.com</font></a>
> <br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
> </font><font size=1 face="sans-serif">bind-users-bounces at lists.isc.org</font>
> <br>
> <hr noshade>
> <br>
> <br>
> <br><font size=3>CATERPILLAR SECURITY ALERT: The email address in the sender
> line does not match the account that sent the email. This can be an indication
> of phishing. Do not click links or open attachments unless you are certain
> it is from a safe source. Learn more at security.cat.com/phishing<br>
> </font>
> <hr><font size=2 face="sans-serif"> We have
> just enabled RPZ with some NSDNAME checks and are seeing an issue resolving
> </font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u></font></a><font si
> ze=2 face="sans-serif">.</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> The first lookup is successful and returns
> both the CNAME and the A record. The second query, within a second
> of the first, will only return the CNAME. It will only return the
> CNAME until the TTL of the A record times out. The first query, when
> it actually has to go out and do recursion will always work. Answering
> from cache will always fail. When you inspect the cache during
> the time that it is only returning the CNAME, the record in cache is "</font><a href=www.wip.rackspace.com><font
> size=2 color=blue face="sans-serif"><u>www.wip.rackspace.com</u></font></a><font size=2 face="sans-serif">
> type ANY NXDOMAIN". This only happens with RPZ's
> enabled and query hitting a RPZ zone with a NSDNAME line. Turning
> off RPZ or whitelisting the lookup via RPZ before it hits a RPZ with NSDNAME
> allows the query to be successful 100% of the time.</font><font size=3>
> <br>
> <br>
> </font><font size=2 face="sans-serif"><br>
> Can anyone else verify this behavior?
> What is going on with </font><a href=www.rackspace.com?><font size=2 color=blue face="sans-serif"><u>www.rackspace.com
> ?</u></font></a><font size=2 face="sans-serif">
> If this is a miss configuration on Rackspace's DNS servers how are
> they not getting hit with support calls like crazy?</font><font size=3>
> <br>
> <br>
> <br>
> </font><font size=2 face="sans-serif"><br>
> dig @redacted.cat.com </font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com<
> /u></font></a><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> @redacted.cat.com </font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u><
> /font></a><font size=3>
> </font><font size=2 face="sans-serif"><br>
> ; (1 server found)</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; global options: +cmd</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; Got answer:</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30337</font><font size=3>
> </font><font size=2 face="sans-serif"><br>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; QUESTION SECTION:</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;</font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u></font></a><font s
> ize=2 face="sans-serif">.
> IN A</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; ANSWER SECTION:</font><font size=3> </font><font size=3 color=blue><u><br>
> </u></font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u></font></a><fon
> t size=2 face="sans-serif">.
> 300 IN CNAME
> </font><a href=www.wip.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.wip.rackspace.com</u></font></a>
> <font size=2 face="sans-serif">.</font><font size=3>
> </font><font size=3 color=blue><u><br>
> </u></font><a href=www.wip.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.wip.rackspace.com</u></font>
> </a><font size=2 face="sans-serif">.
> 30 IN A
> 173.203.44.116</font><font size=3> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; Query time: 193 msec</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; SERVER: redacted</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; WHEN: Wed May 7 08:53:08 2014</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; MSG SIZE rcvd: 73</font><font size=3> <br>
> <br>
> <br>
> </font><font size=2 face="sans-serif"><br>
> dig @redacted.cat.com </font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com<
> /u></font></a><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>>
> @redacted.cat.com </font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u><
> /font></a><font size=3>
> </font><font size=2 face="sans-serif"><br>
> ; (1 server found)</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; global options: +cmd</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; Got answer:</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25905</font><font size=3>
> </font><font size=2 face="sans-serif"><br>
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; QUESTION SECTION:</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;</font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u></font></a><font s
> ize=2 face="sans-serif">.
> IN A</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; ANSWER SECTION:</font><font size=3> </font><font size=3 color=blue><u><br>
> </u></font><a href=www.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.rackspace.com</u></font></a><fon
> t size=2 face="sans-serif">.
> 298 IN CNAME
> </font><a href=www.wip.rackspace.com><font size=2 color=blue face="sans-serif"><u>www.wip.rackspace.com</u></font></a>
> <font size=2 face="sans-serif">.</font><font size=3>
> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; AUTHORITY SECTION:</font><font size=3> </font><font size=2 face="sans-serif"><br>
> wip.rackspace.com. 58 IN
> SOA www-gtm-ord1.rackspace.com. hostmaster.305181-GTM1.rackspace.com.
> 86 10800 3600 604800 60</font><font size=3> <br>
> </font><font size=2 face="sans-serif"><br>
> ;; Query time: 2 msec</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; SERVER: redacted</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; WHEN: Wed May 7 08:53:10 2014</font><font size=3> </font><font size=2 face="sans-serif"><br>
> ;; MSG SIZE rcvd: 129</font><font size=3> <br>
> <br>
> </font><font size=5 color=blue><b><br>
> David A. Evans</b></font><font size=3> <b><br>
> Enterprise IP/DNS Management</b> <b><br>
> Network Infrastructure Tools and Services</b> </font><font size=3 color=blue><u><br>
> </u></font><a href=mailto:Evans_David_A at cat.com><font size=3 color=blue><b><u>Evans_David_A at cat.com</u></b></font></a>
> <font size=3>
> </font><tt><font size=2>_______________________________________________<br>
> Please visit </font></tt><a href="https://lists.isc.org/mailman/listinfo/bind-users"><tt><font size=2>https://lists.is
> c.org/mailman/listinfo/bind-users</font></tt></a><tt><font size=2>
> to unsubscribe from this list<br>
> <br>
> bind-users mailing list<br>
> bind-users at lists.isc.org<br>
> </font></tt><a href="https://lists.isc.org/mailman/listinfo/bind-users"><tt><font size=2>https://lists.isc.org/mailman
> /listinfo/bind-users</font></tt></a>
> <br>
> --=_alternative 0054318186257CD1_=--
>
> --===============1900647751663457684==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
> --===============1900647751663457684==--
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka at isc.org
More information about the bind-users
mailing list