High loads...

[Bill Davidsen]

Russ Allbery wrote:
> Bjorn Hellqvist <bjhell at telia.net> writes:
>   
>> On Wed, 14 Mar 2007, Russ Allbery wrote:
>>     
>
>   
>>> I'm definitely missing something since no matter what you're making
>>> your single client do, it shouldn't be talking to more than a handful
>>> of nnrpd processes at most.  Which doesn't explain what 95 processes
>>> are all in the run queue on your system.  INN doesn't usually spawn
>>> that many processes due to its architecture unless you have a bunch of
>>> readers, which you say that you don't.
>>>       
>
>   
>>> What does top say is trying to run?
>>>       
>
>   
>> nnrpd for the most parts...
>>     
>
>   
>> This is the output of news.notice when its at the top.
>>     
>
>   
>> "Unable to allocate memory for transaction detail"
>>     
>
>   
>> What memory are reported to be out of?
>>     
>
> System memory.  You have enough nnrpds running that you probably ran out
> of swap.  (The reason why your system is so slow and has such a high load
> is that you ran the system into swap, most likely.)
>
>   
>> Maybe this is not a real problem on real use. I am currently pushing
>> 20-30 new connections per second to the system. And are doing this for a
>> couple of minutes. ( 16:06:35 - 16:09:09 )
>>     
>
> If you're starting 25 new connections every second for about 150 seconds,
> at the end of that you have 3,750 nnrpd processes running (assuming that
> you're not closing any of those connections down).  If each one has 1MB of
> overhead memory required, you need 3.7GB of system memory to be able to
> run them all plus whatever innd needs.
>
> INN can handle that sort of load given appropriate resources, but the
> resources may be high.
>
>   
>> But never the less. I dont want a operational system to die out like
>> that from just one abusive customer.
>>     
>
> So what you really want to do is prevent someone from opening that many
> connections that quickly, since nearly 4,000 simultaneous users probably
> doesn't represent your real production load.  For that, either run innd
> and nnrpd on separate ports (433 is the recommended choice for the port
> line in inn.conf) and then run nnrpd under xinetd or some other smart
> server that can do connection throttling, or look into using the -W, -T,
> and -X flags to innd (set via innflags in inn.conf and documented in the
> innd man page).  Just using -X 30 or thereabouts would probably keep this
> problem under control.
>
>   
I half-agree with Russ, running on separate ports is the way to go. 
However, rather than letting xinetd start a separate process for each 
connection, I always ran nnrpd as a daemon (-D) when I ran the servers 
at Prodigy/SBC. This appeared to give a lower impact for process 
initiate, since there was just a fork rather than a fork+exec. I don't 
have the figures any more, I was careful not to take anything with me, 
but memory usage was lower in real memory, presumably due to copy on 
write. That was Linux, YMMV.

All the stuff about flags and options is what you need to study, and 
pick values which keep you machine from melting.

-- 
bill davidsen <davidsen at tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979