Rewriting INN's build system

[Russ Allbery]

Currently, INN is using a hand-rolled build system that has remnants of
the results of the original Autoconf conversion from the old INN build
system.  It was designed in part for requirements from around 2000, such
as adjusting for the fact that libtool was painfully slow on some systems.
It also used some ideas that I was using for other packages around that
time.

Since then, I've learned lots more about build systems, and have developed
a fairly standardized build system design that I'm using on all my other
projects, but which INN never benefited from since I hadn't had much time
to work on INN.  Among other things, after years of hating Automake, I
finally came around to the amount of time it saves, particularly if you
use it in non-recursive mode.  I'm also using libtool uniformly elsewhere,
and I don't think the speed issues are still particularly important.

I'm also now maintaining a separate rra-c-util package that collects
portability code and utility functions that I use across multiple
packages, which started life as some parts of INN's library plus code that
I wrote for INN but has expanded considerably since then.  Julien recently
did a massive resync with that code, but it's harder than it is for my
other packages because INN isn't laid out the same way, which causes
includes to change and various other things that require adjustments that
can be a bit tedious.

I'd therefore like to propose pretty radically overhauling the INN build
system to look like my other projects: non-recursive Automake, separate
portable and util directories, renaming some things to match rra-c-util's
naming conventions, always using libtool in the build, and so forth.  I
think this would make the build system both more robust and more like many
other free software projects, and will also make it easier to share code
with rra-c-util.

I think I might even have time to do this within the next couple of
months, if people think it's a good idea.

Clear benefits:

* We gain the benefits of all new Automake development.  This includes
  things like optional silent builds (one line per built file), which I
  wouldn't turn on by default but which is very nice during development
  builds so that one can see the warnings without a clutter of other
  messages, portability to make without the type of includes that we
  assume, automatic dependency handling with no make depend, and the
  ability to use things like Automake conditionals to simplify the
  makefiles.

* Non-recursive make gives us a correct global dependency tree, which
  means everything is rebuilt properly, every time.  No more editing a
  file in lib and then running make in innd and having nothing change
  because the innd Makefile doesn't realize that the file modification
  means libinn needs to be rebuilt.

* Proper support for builds outside of the source tree or with a read-only
  source tree.

* We would stop using libtool in a weird conditional way, which is
  currently causing problems because that's never tested and tends to
  break.  (Updating libtool right now causes errors from sed in
  configure.)

* Better separation of the INN library for doing INN things, portability
  code so that INN itself will build elsewhere, and utility functions that
  we may or may not want to export to clients of the INN library.

* A much easier path to start with proper shared library discipline with
  symbol versioning and controlling the shared library exports.

* Much easier to share code with rra-c-util, which is in use by nine other
  projects and counting and hence is tested on a wider variety of systems
  and in a wider variety of circumstances than INN may be at the moment.

* Could stop manually importing libtool and config.{guess,sub} and just
  install them in the tree using the autogen script, which means carrying
  around fewer external files in the source tree.

Things that I think are improvements but others may be surprised by:

* Generate man pages and text documentation from POD in autogen rather
  than as part of the build with Makefiles.  This is needed for proper
  support of building outside the source tree.  It's a bit more tedious if
  only one file has changed, but I do this now with other projects and I
  think it's cleaner.

* Eliminate MANIFEST and instead use Automake's make dist support.  This
  means fewer support scripts and idiosyncratic ways of doing things, but
  it does mean we won't have the file documenting what every file is for
  (although I'm not sure how really useful that is).

* The various make variables and environment variables would have more
  standard meanings.  This means things like COPT would go away, setting
  CFLAGS from the command line would work, and so forth.

* Lots of code reorganization around the build system, which means that
  stuff would be in new directories and new places.

Drawbacks:

* We'd lose the backup .OLD support for updated binaries and similar files
  on make upgrade.  I'm not sure what to think about this.  We could try
  to hack it into Automake, I suppose, but it's definitely a bunch of
  extra complexity.  Could we just tell people to make a backup of their
  install tree before running make upgrade?  How much do people actually
  use this?

* Non-recursive make means that you can't just run make in the directory
  you're working in.  make only works at the top level.  You can build a
  single program, but the syntax is a bit more tedious (you have to
  specify the path of the file to build on the make command line).

* We'd lose the modular plug-in build system for history, storage, and
  overview methods.  Or, rather, we wouldn't have to, but I'd not bother
  to convert that and would replace it with straightforward Automake rules
  for the in-tree methods, since I don't think the complexity is worth it.
  It's a really neat idea, but I don't think there are a bunch of
  alternative backend developers out there who are actively using the
  ability to drop their code into INN's tree but don't want us to just
  check it into CURRENT.

* libtool used on every build, so builds will be somewhat slower.

* Lots of churn relative to STABLE, which will make backporting changes
  somewhat more difficult, particularly if they have any build
  implications.  There would be a lot of changes to the naming of include
  files, which would touch basically every file.

* I would propose leaving the files imported from rra-c-util verbatim, but
  the coding style is slightly different than INN's.  (For INN, I used a
  hybrid of my style and INN's native style, particularly around block
  comments, and my own is somewhat different.)  This means that some files
  in INN (although ones segregated in their own directories for the most
  part) would have a slightly different style.

What do people think?

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.