How to Trace "TCP Receive Error"

[Barry Finkel]

On 6-Jan-08, at 11:05 AM, Barry Finkel wrote:

>>>> I am seeing lots of messages like this one from BIND-9.4.1-P1:
>>>>
>>>>     [ID 873579 daemon.info] dispatch b090ef8:
>>>>       shutting down due to TCP receive error: 69.59.189.68#53:
>>>>       connection reset


Mark Andrews replied,

>>> I suspect the nameserver has some sort of filtering box in
>>> front of it that is attempting to determine if the client
>>> is real or spoofed.  A "real" client will try TCP on seeing
>>> "tc" even if this is not strictly true for UDP only
>>> client/stacks.  This then turns just about all the UDP
>>> queries into TCP queries.  If the nameserver behind gets
>>> overwhelmed with TCP connections it will start sending out
>>> RST.  Self inflicted TCP SYN DoS.  There is a reason DNS
>>> uses UDP in the first place.

>>> This sort of "solution" does not scale.

>>> From the trace below the filtering box is keeping state
>>> because subsequent UDP queries get through.  This doesn't
>>> help much as many clients only ask a single question of a
>>> nameserver as A and AAAA queries often go to different
>>> nameservers.  If the filtering boxes were to share state
>>> there would be less problems.


At 18:58 06-01-2008, Barry Finkel replied:
>>A check of our BIND query log shows lots of queries from one of our
>>mail machines; here is one query.
>>
>>      06-Jan-2008 17:38:01.101 queries: info:
>>        client 146.137.96.51#41548: query:
>>        achilles.ctd.anl.gov.dob.sibl.support-intelligence.net IN A +
>>
>>I do not have access to that mail machine, so I am copying the
>>administrators of that machine, who might be able to tell me why these
>>queries are happening.


And SM replied to me off-list:
>dob.sibl.support-intelligence.net is a blacklist.  These queries are 
>most probably generated by SpamAssassin when mail is scanned.  There 
>are reports of similar problems for DNS queries against that blacklist.
>
>Regards,
>-sm 


I have gotten a snoop trace, and I have reviewed it.  A UDP query

     Address(19.36.221.140.dob.sibl.support-intelligence.net.)

(packet #6) gets a return UDP packet (#11) with

     AA, TC, 0 answer sections, 0 auth sections, and 0 additional sects.

There is truncation involved.  In the trace, I see these TCP packets
following the TC UDP packet:

	     oberon ==> dob.sibl.support-intelligence.net.
	     oberon <== dob.sibl.support-intelligence.net.
     P# Time
     -- ----------- --- --------------------------------------
     14 08:03:16.18 ==> (DNS) syn       3389810912          0
     21 08:03:16.24 <== (DNS) ack syn   984205940  3389810913
     22 08:03:16.24 ==> (DNS) ack       3389810913 984205941
     31 08:03:16.29 <== (DNS) ack reset 984205941  3389810913

The dob server resets the TCP connection very quickly.
The trace is confusing, as my DNS server oberon has sent five queries
in a short period of time (packets 6-10, all at 08:03:16.13), and so
there are five successive TC responses and then five different TCP
streams to the dob nameserver.  I think I have extracted the correct
TCP packets in the four I have reduced above.

A Google search of the dob domain retrieves web pages with similar
complaints.
----------------------------------------------------------------------
Barry S. Finkel
Computing and Information Systems Division
Argonne National Laboratory          Phone:    +1 (630) 252-7277
9700 South Cass Avenue               Facsimile:+1 (630) 252-4601
Building 222, Room D209              Internet: BSFinkel at anl.gov
Argonne, IL   60439-4828             IBMMAIL:  I1004994