How does Yahoo/Google find unknown domains?

Lyle Giese lyle at
Wed Nov 10 01:35:49 UTC 2010

Michelle Konzack wrote:
> Hello Robert Spangler,
> Am 2010-11-09 10:34:52, hacktest Du folgendes herunter:
>> If these domains are for internal use only, why did you list the DNS servers 
>> for them?  You are aware that you can register a domain without listing a DNS 
>> Server?
> Because my own customers (exclusively) must access it.  They are my VOIP
> and IPTV servers and there is no public HTTP content  but  I  am  bombed
> with PHP/CGI requests and I do not know, where Google and Co have gotten
> those links.
Are you sure it's all search bots and not script kiddies playing? 
Having a valid index.html in place that redirects to your business home
page may help.  But the search bots don't normally search around for
php/cgi scripts.  That however is typical of the script kiddies looking
for a server to hack.
> Some of the searchbots are hiting my servers  3  times  in  series  from
> three different IPs and in summary, I have more then  10.000  searchbot-
> entries per day in my Logs.
> My server always return an Error-Page from the VServer that there is  no
> configured HTTP host on the machine but it is ignored.  One  VHost  must
> be configured for the web administration and it is hit too and too much!
> Even my simple squirrelmal login page from <>  is
> spidered daily with more then 800 hits and I have already  counted  more
> then 80 different searchbots.
> How braindamaged are Searchbot-Programmers?
> All of my webservers together have arround 86 TByte of content including
> a VERY huge debian archive (all releases and versions from 0.96 to  now)
> and my traffic per month is arround 27 TByte.
> The searchbots  are  creating 17,3 TByte  traffic  per  month  which  my
> customers have to pay to!
> Maybe I call my lawer to write letters to the serachbot owners  to  stop
> spidering my 36 domains.
> Oh, at Level3 in Frankfurt I pay 12 Euro/Mbit traffic  per  month  which
> mean 12 Euro per 320 GByte traffic. Not counting the price for the 700km
> FiberOptic line which is another provider (0,40 euro/m/year).
> I had a 1 GE line from Frankfurt but du to the excessiv serchbot traffic
> it broke several times per day.  Now I have in total twelf 1GE  (Level3,
> Verizon, DTag and  Orange).  Maximal  I  can  have  64 x 10 GE  with  my
> Transmode TS System but then I can install my own BPOP.
> Thanks, Greetings and nice Day/Evening
>     Michelle Konzack
Despite how I feel about Yahoo's SLURP engine, it still honors
robots.txt.  Script kiddies don't.

Lyle Giese
LCR Computer Services, Inc.

P.S.  My last post on this.  This is not DNS related.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the bind-users mailing list