How does Yahoo/Google find unknown domains?
stenc at s-carlsen.dk
Mon Nov 8 01:32:14 UTC 2010
Did you consider robots.txt? Well behaved spiders should respect that,
although it does not prevent anything.
I would expect google and yahoo to respect it, I have no clue about he.net.
On 08/11/10 2:02, Michelle Konzack wrote:
> Hello experts and *,
> I have (since several years) collected some domain names which do not
> exist (since years) and registered it in the last 4 month for the
> internal use of my Internet Service.
> Now I see Googlebot, Yahoo and <he.net> quering my DNS Servers for
> exactly those domains.
> If I read the conditions of Networksolutions and Co, spidering of WHOIS
> records is prohibited also the commercial use of the data.
> Does someone have an experience with his crap?
> Unfortunately I can not deny access to the 180 servers and Google, Yahoo
> and He is bombing my network with to much useless requests. I have
> written a mail to Google not to attack my network of VOIP and IPTV
> servers, but they continue...
> The webservers have only an SHTTP administrativ VHost, but not <exp.com>
> or <www.exp.com> but the webserver get any requests from <*.exp.com>
> because it is an administrative VServer and the error logfile is per day
> VERY long.
> An htaccess does not work, because I have more then 800 VHosts on each
> Thanks, Greetings and nice Day/Evening
> Michelle Konzack
> bind-users mailing list
> bind-users at lists.isc.org
No improvements come from shouting:
"MALE BOVINE MANURE!!!"
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the bind-users