batch reverse dns lookup

[Guido Roeskens]

Hello,

Danny Mayer wrote:
> At 12:36 PM 6/12/2004, Andy Peters wrote:
> 
> 
>>Hi i don't know if this is quite the group to ask this in (you are the dns
>>guru's ?) so feel free to offer alternative group suggestions
>>
>>here's the problem, i have a list of domains (26,000+) in plain text format
>>(its a hosts file to be exact) and i would like to seperate the domains that
>>resolve to an ip from the ones that don't (to clean up the list and remove
>>invalid and dead entries)

how do you want to know what entries are dead or invalid?
Let's assume the file is a list of all names which are/were used for a
web hosting company.

On a high level you could do the following:
- generate a list of all ip's and or names
   (I would use a hash of hash in perl; the following line
    172.17.233.55 www.domain1.com info.domain2.org
    would be made into $myhash{$ip}{$name}
    --- snip ---
    ($ip, @names) = split( $line );
    $myhash{$ip}{$_} = 0 foreach (@names);
    --- snip ---
- ping all ips (but beware, ICMP may be disallowed or hosts may be down)
- lookup what names in the hash for an ip exist in DNS
   * if the name exists, does the name point to the ip in the hash
     (there can be more than one ip for a name; maybe a name is delegated
      not an A record but one or more NS records)
- is the name delegated to your dns servers
   * cut of first part of the name and look at the NS set
     are your DNS servers listed as servers?
     (you may have to look at higher levels also; for www.sub1.domain.com
      you may also need to look at `dig domain.com NS` )
- for web servers you could use something like a telnet expect script
   or in perl:
   --- snip ---
   # put this on top of script
   use LWP::UserAgent;

   my $ua  = LWP::UserAgent->new;
   my $req = new HTTP::Request 'GET' => "$name";
   # make the request to the server
   my $res = $ua->request($req, $tmp_in);
   if ( $res->is_error() ) {
	# there was an error, so this may be a stale entry
	# if we got a 30x response, the name may be valid
	# you need to check this yourself
   }
   # no error so the name seems to be valid
   #  ....
   --- snip ---

- then you need to correlate all the data gathered and try to figure out
which are valid and which aren't
- maybe you would also need to check for mail servers, name servers and
so on....

>>any idea how i might achieve it ? i would of thought a quick bash or batch
>>script could do this but not having any experience in shell scripts i dont
>>know how i could do this
>>any help or pointers would be appreciated (i have access to
>>suse/knoppix/win*/)
>>
>>cheers folks
>>
>>Andy P
> 
> 
> Don't bother. Just delete the hosts file. You don't need it as long as you 
> have DNS
> and the machine points to some nameservers. You are just wasting your time.
You don't know what he is trining to acheive.
Maybe this file is some sort of "user database". In the above example of
a web hoster there might be all virtual hosts in this file.

In the O'Reilly DNS and Bind books there are scripts to convert a hosts
file into zone files. Maybe these (this) script(s) were used to convert
the flat file into zone files.

You could download the scripts from the O'Reilly website and run them on
the file. Compare the output of the script(s) to existing zone files.
> Worse than that the hosts file gets read sequentially (usually) and it will 
> take
> forever to find the name it needs. That's why DNS was designed in the first 
> place.
> 
> Danny
> 

Good luck in cleaning up....


Guido