Massive increase of SERVFAIL after April 28th 2025.

[Carlos Horowicz]

Hi Vincent

using a conventional resolver (no rpz, no forwards, no forward zones) 
from our Miami cloud:


Tracing to ftp.lip6.fr[a] via 190.185.104.10, maximum of 3 retries
190.185.104.10 (190.185.104.10)
  |\___ g.ext.nic.fr [fr] (2001:0678:004c:0000:0000:0000:0000:0001)
  |     |\___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) Got authoritative 
answer [received type is cname]
  |     |\___ isis.lip6.fr [lip6.fr] (132.227.60.2) Got authoritative 
answer [received type is cname]
  |      \___ osiris.lip6.fr [lip6.fr] (132.227.60.30) Got authoritative 
answer [received type is cname]
  |\___ g.ext.nic.fr [fr] (194.0.36.1)
  |     |\___ osiris.lip6.fr [lip6.fr] (132.227.60.30) (cached)
  |     |\___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) (cached)
  |      \___ isis.lip6.fr [lip6.fr] (132.227.60.2) (cached)
  |\___ d.nic.fr [fr] (2001:0678:000c:0000:0000:0000:0000:0001)
  |     |\___ isis.lip6.fr [lip6.fr] (132.227.60.2) (cached)
  |     |\___ osiris.lip6.fr [lip6.fr] (132.227.60.30) (cached)
  |      \___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) (cached)
  |\___ d.nic.fr [fr] (194.0.9.1)
  |     |\___ isis.lip6.fr [lip6.fr] (132.227.60.2) (cached)
  |     |\___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) (cached)
  |      \___ osiris.lip6.fr [lip6.fr] (132.227.60.30) (cached)
  |\___ f.ext.nic.fr [fr] (2001:067c:1010:0011:0000:0000:0000:0053)
  |     |\___ osiris.lip6.fr [lip6.fr] (132.227.60.30) (cached)
  |     |\___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) (cached)
  |      \___ isis.lip6.fr [lip6.fr] (132.227.60.2) (cached)
   \___ f.ext.nic.fr [fr] (194.146.106.46)
        |\___ osiris.lip6.fr [lip6.fr] (132.227.60.30) (cached)
        |\___ isis.lip6.fr [lip6.fr] (132.227.60.2) (cached)
         \___ soleil.uvsq.fr [lip6.fr] (193.51.24.1) (cached)

osiris.lip6.fr (132.227.60.30)          ftp.lip6.fr -> nephtys.lip6.fr
osiris.lip6.fr (132.227.60.30)          nephtys.lip6.fr -> 132.227.74.17
isis.lip6.fr (132.227.60.2)             ftp.lip6.fr -> nephtys.lip6.fr
isis.lip6.fr (132.227.60.2)             nephtys.lip6.fr -> 132.227.74.17
soleil.uvsq.fr (193.51.24.1)            ftp.lip6.fr -> nephtys.lip6.fr
soleil.uvsq.fr (193.51.24.1)            nephtys.lip6.fr -> 132.227.74.17

HTH

Carlos Horowicz
Planisys


On 01/05/2025 18:07, vincent at cojot.name wrote:
>
> Hi Carlos,
>
> First of all, I'd like to say how sorry I was for those affected, as I 
> was watching the events unfold down south.
>
> I've rebuilt dnstracer for RHEL9 and I don't really understand what's 
> going on here.. Here's the output for ftp.lip6.fr:
>
> # dnstracer -q cname -s M.GTLD-SERVERS.NET  ftp.lip6.fr
> Tracing to ftp.lip6.fr[cname] via M.GTLD-SERVERS.NET, maximum of 3 
> retries
> M.GTLD-SERVERS.NET (2001:0501:b1f9:0000:0000:0000:0000:0030) Refers 
> backwards
>
> Same output from any of my bind hosts:
>
> # dnstracer -q cname -s 127.0.01  ftp.lip6.fr
> Tracing to ftp.lip6.fr[cname] via 127.0.01, maximum of 3 retries
> 127.0.01 (127.0.0.1) Refers backwards
>
> But interestingly, doing this with www.google.com instead of 
> ftp.lip6.fr -only- works on the bind servers with forwarders 
> configured. On a test bind host without the forwarders, I get this:
>
> # dnstracer -q cname -s 127.0.01  www.google.com
> Tracing to www.google.com[cname] via 127.0.01, maximum of 3 retries
> 127.0.01 (127.0.0.1) Refers backwards
>
> Vincent
>
> On Thu, 1 May 2025, Carlos Horowicz via bind-users wrote:
>
>>
>> Hi,
>>
>> For SERVFAIL to happen, ALL authoritative for the affected domains 
>> must have been in Datacenters in Spain, Portugal or southern France.
>>
>> I live in Spain, and as 12:33 CET I lost not only power but basic 
>> telephony, cellular telephony and cellular data. Everything. Power 
>> generators were only good for keeping power
>> locally at Datacenters or Hospitals, but they were isolated from each 
>> other.
>>
>> The mitigation began at around 2-3pm CET , as they were turning up 
>> different power plants one at a time and connecting it to the power 
>> network, and it took them more than 12
>> hours to turn everything up.
>>
>> So may be that was the reason, if it coincides with your perception 
>> ... dnstracer has eventually helped me find lame delegations.
>>
>> Carlos Horowicz
>> Planisys
>>
>> On 01/05/2025 17:23, Rob McEwen via bind-users wrote:
>>       From vincent at cojot.name
>>       until a few days ago (April 28th?) when the amount of SERVFAIL 
>> started going ballistic and started preventing the resolution of a 
>> lot of DNS names on the
>>       internet to the point where DNS was unusable
>>
>>
>> I strongly suspect that this was caused (even if indirectly?) by the 
>> MASSIVE and many-hours-long power outages in Europe, mainly in Spain 
>> and Portugal. That started on
>> April 28, 2025, at approximately 6:33 a.m. Eastern Time (ET) - and 
>> the majority of it lasted almot 24 hours.
>>
>> https://www.france24.com/en/europe/20250430-what-we-know-so-far-about-the-massive-blackout-that-hit-spain-and-portugal 
>>
>>
>> Hopefully, you're not seeing any more of these errors now?
>>
>> Rob McEwen, invaluement
>>
>>
>>
>>