How does Yahoo/Google find unknown domains?
lyle at lcrcomputer.net
Wed Nov 10 01:35:49 UTC 2010
Michelle Konzack wrote:
> Hello Robert Spangler,
> Am 2010-11-09 10:34:52, hacktest Du folgendes herunter:
>> If these domains are for internal use only, why did you list the DNS servers
>> for them? You are aware that you can register a domain without listing a DNS
> Because my own customers (exclusively) must access it. They are my VOIP
> and IPTV servers and there is no public HTTP content but I am bombed
> with PHP/CGI requests and I do not know, where Google and Co have gotten
> those links.
Are you sure it's all search bots and not script kiddies playing?
Having a valid index.html in place that redirects to your business home
page may help. But the search bots don't normally search around for
php/cgi scripts. That however is typical of the script kiddies looking
for a server to hack.
> Some of the searchbots are hiting my servers 3 times in series from
> three different IPs and in summary, I have more then 10.000 searchbot-
> entries per day in my Logs.
> My server always return an Error-Page from the VServer that there is no
> configured HTTP host on the machine but it is ignored. One VHost must
> be configured for the web administration and it is hit too and too much!
> Even my simple squirrelmal login page from <webmail.tamay-dogan.net> is
> spidered daily with more then 800 hits and I have already counted more
> then 80 different searchbots.
> How braindamaged are Searchbot-Programmers?
> All of my webservers together have arround 86 TByte of content including
> a VERY huge debian archive (all releases and versions from 0.96 to now)
> and my traffic per month is arround 27 TByte.
> The searchbots are creating 17,3 TByte traffic per month which my
> customers have to pay to!
> Maybe I call my lawer to write letters to the serachbot owners to stop
> spidering my 36 domains.
> Oh, at Level3 in Frankfurt I pay 12 Euro/Mbit traffic per month which
> mean 12 Euro per 320 GByte traffic. Not counting the price for the 700km
> FiberOptic line which is another provider (0,40 euro/m/year).
> I had a 1 GE line from Frankfurt but du to the excessiv serchbot traffic
> it broke several times per day. Now I have in total twelf 1GE (Level3,
> Verizon, DTag and Orange). Maximal I can have 64 x 10 GE with my
> Transmode TS System but then I can install my own BPOP.
> Thanks, Greetings and nice Day/Evening
> Michelle Konzack
Despite how I feel about Yahoo's SLURP engine, it still honors
robots.txt. Script kiddies don't.
LCR Computer Services, Inc.
P.S. My last post on this. This is not DNS related.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the bind-users