How does Yahoo/Google find unknown domains?

Wed Nov 10 01:14:36 UTC 2010

On Wed, Nov 10, 2010 at 01:47:44AM +0100,
 Michelle Konzack <linux4michelle at tamay-dogan.net> wrote 
 a message of 115 lines which said:

> Even my simple squirrelmal login page from <webmail.tamay-dogan.net>
> is spidered daily with more then 800 hits and I have already counted
> more then 80 different searchbots.

HTTP spidering issues are clearly off-topic for this list, which deals
with the BIND DNS server. I suggest to ask on ServerFault or a similar
forum.

> The searchbots are creating 17,3 TByte traffic per month which my
> customers have to pay to!

There is no other solution than robots.txt.