"the problem with threads"

Danny Mayer mayer at gis.net
Mon Sep 11 21:49:08 UTC 2006


Brad Knowles wrote:
> At 4:56 PM +0000 2006-09-07, Paul Vixie quoted Shane Kerr:
> 
>>>  - - All concurrent programs are provably bad
>>
>>  that's not the way i read this article.  but speaking from bind9's
>> experience,
>>  actually getting useful work out of N processors is very much more
>> difficult
>>  than "use threads".
> 
> Parallelism is hard.  Closely coupled IPC is hard.  The multiprocessor
> super computer guys can talk about those problems at length.  There are
> a surprising number of problems that do not parallelize very well, and I
> don't think it matters too much what your programming model is, although
> some do make solving this kind of problem harder than others do.
> 

The super computer guys merely move the line, they don't solve the
problem, that's what the programmers have to do.

> I think the mistake that a lot of people make is that they blindly throw
> solutions at a problem to see what sticks, and these solutions
> frequently amount to whatever the latest fad is to patch some old
> programming method.
> 

Agreed, but then new fads tend to be not well documented or understood
and it's easier to fall back on a known and more-or-less-understood method.

> I am not convinced that threads are inherently bad.  I am convinced that
> they are hard for many people to use, and very hard for most people to
> use correctly -- even in those situations where they are actually
> appropriate.

They are not inherently bad, but they can be difficult to implement.
Some implementations of threads were also not done right so you had the
situation where BIND 9 could be built multithreaded but it was not
recommended because of problems with the threads package. That hardly
inspires confidence in threading models. Done right, the hard part is
still figuring out the locking to prevent either deadlocks between
competing threads, and race conditions and other bizarre behavior.
Locking is very expensive yet you want to lock a memory structure for as
short a time as possible in order to reduce lock contention to keep the
threads busy. It's the locking more than anything that prevents you
scaling with the addition of multiprocessors.

> 
> Of course, I'm not qualified to tell you which kinds of situations fall
> into which categories, or which type of people fall into which
> categories.  I'm just extrapolating from what I've seen in the various
> projects I've been involved with, and multithreaded programming seems to
> be something that trips up a hell of a lot of people, even a lot of
> people that I would otherwise consider to be very good programmers.
> 

Yes, it's not simple and flawed packages make things more difficult.
> 
> And this multicomputing problem just gets worse, as modern processors
> and operating systems get more complex.
> 

Not to mention that the O/S's themselves use multithreading to try and
take advantage of the multiprocessors.

>>>  - - We don't really know what else to do
>>
>>  i think the VLIW people believe that they know what else we should do
>> :-).
>>  i do not myself know what else we should do, unless it's the apache
>> fork()
>>  model.
> 
> I've been a fan of the KISS principle for a long time, and I think the
> apache/postfix approach of having multiple cooperating
> intercommunicating processes (which are also mutually distrustful and
> operate at least possible privilege) may be one of the best real-world
> compromises that I have come across.
> 

That too is hard to implement.

> However, I'm not sure that I understand how even this model can be
> applied to many different types of problems, especially ones like those
> faced by named or ntpd, both of whom have to operate on potentially
> forged UDP packet data, and need to operate in a manner somewhat like
> many near real-time programs.
> 

Well you validate the packet first before you consume it.

> 
> At this point, I can recognize some issues, and I can recognize some
> models that I have encountered which seemed to work at least reasonably
> well in certain circumstances in the past, but that's about the best I
> can do.
> 

There are lots of models and even languages to address the issue but
they all boil down to the need at some point in the process of having to
share information with other pieces and something needs to wait.

> I was hoping that you folks would have some answers for me.
> 

Probably not likely I'm afraid.

Danny


More information about the bind-workers mailing list