How does Yahoo/Google find unknown domains?

Sten Carlsen stenc at
Mon Nov 8 01:32:14 UTC 2010

Did you consider robots.txt? Well behaved spiders should respect that,
although it does not prevent anything.

I would expect google and yahoo to respect it, I have no clue about

On 08/11/10 2:02, Michelle Konzack wrote:
> Hello experts and *,
> I have (since several years) collected some domain names  which  do  not
> exist (since years) and registered it  in  the  last  4  month  for  the
> internal use of my Internet Service.
> Now I see Googlebot, Yahoo and  <>  quering  my  DNS  Servers  for
> exactly those domains.
> If I read the conditions of Networksolutions and Co, spidering of  WHOIS
> records is prohibited also the commercial use of the data.
> Does someone have an experience with his crap?
> Unfortunately I can not deny access to the 180 servers and Google, Yahoo
> and He is bombing my network with  to  much  useless  requests.  I  have
> written a mail to Google not to attack  my  network  of  VOIP  and  IPTV
> servers, but they continue...
> The webservers have only an SHTTP administrativ VHost, but not <>
> or <> but the webserver get  any  requests  from  <*>
> because it is an administrative VServer and the error logfile is per day
> VERY long.
> An htaccess does not work, because I have more then 800 VHosts  on  each
> server.
> Thanks, Greetings and nice Day/Evening
>     Michelle Konzack
> _______________________________________________
> bind-users mailing list
> bind-users at

Best regards

Sten Carlsen

No improvements come from shouting:


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the bind-users mailing list