Meditation, The Art of Exploitation

Thinking? At last I have discovered it--thought; this alone is inseparable from me. I am, I exist--that is certain. But for how long? For as long as I am thinking. For it could be, that were I totally to cease from thinking, I should totally cease to exist....I am, then, in the strict sense only a thing that thinks.

Monday, February 18, 2008

makefile, dependencies, and single target multiple rules

To work with a large software system, a reliable and fast build system is one of the most important pieces. Without a reliable and fast build system, development can ground to a halt. Imagine if it takes 10-15 minutes to build a new version of the software when there were one or two changes. Unfortunately this happens more often than one might have expected. GNU make provides a framework to construct a build system through makefiles. Most of us use it on a daily basis.

There is one important and interesting feature of makefile often overlooked, automatic dependency generation and inclusion. This feature is enabled by the fact that gmake allows a single target having multiple dependences but only one rule can have associated action. (Rules, dependencies, it's like lex/yacc, isn't it?)

objects = foo.o bar.o
foo.o : defs.h
bar.o : defs.h test.h
$(objects) : config.h

In this quoted example, foo.o depends on both defs.h and config.h.

Thus automatic dependency is typically done like this:

foo.o: f1.c f2.h
gcc -c f1.c
gcc -E -M -MF dep/foo.d -c f1.c
-include dep/foo.d

in this example, a dependency file dep/foo.d is generated and included in the makefile. Initially the include file does not exist and an error will be reported without '-' before include. '-' suppresses error reporting in gmake commands. The first time gmake runs through this makefile with 'make dep', it will generate the dependency file and compile f1.c. Alternatively, one can force the dependency generation as in the following example with 'make' (if all is the first rule in this makefile) but since its effect is not immediately available in compiling foo.o it's generally considered a bad practice. It forces generation of dependencies files every time during 'make all' (this can be somewhat alleviated by using make or shell conditionals). Unless a user of the makefile has a good reason to do so, avoid it:

all: dep foo.o
foo.o: f1.c f2.h
gcc -c f1.c
gcc -E -M -MF dep/foo.d -c f1.c
-include dep/foo.d

The various flags used in this example taken from gcc manual:

-E Stop after the preprocessing stage; do not run the compiler proper.
The output is in the form of preprocessed source code, which is
sent to the standard output.

Input files which don't require preprocessing are ignored.

-M Instead of outputting the result of preprocessing, output a rule
suitable for make describing the dependencies of the main source
file. The preprocessor outputs one make rule containing the object
file name for that source file, a colon, and the names of all the
included files, including those coming from -include or -imacros
command line options.

Unless specified explicitly (with -MT or -MQ), the object file name
consists of the basename of the source file with any suffix
replaced with object file suffix. If there are many included files
then the rule is split into several lines using \-newline. The
rule has no commands.

This option does not suppress the preprocessor's debug output, such
as -dM. To avoid mixing such debug output with the dependency
rules you should explicitly specify the dependency output file with
-MF, or use an environment variable like DEPENDENCIES_OUTPUT.
Debug output will still be sent to the regular output stream as

Passing -M to the driver implies -E, and suppresses warnings with
an implicit -w.

-MF file
When used with -M or -MM, specifies a file to write the dependen-
cies to. If no -MF switch is given the preprocessor sends the
rules to the same place it would have sent preprocessed output.

When used with the driver options -MD or -MMD, -MF overrides the
default dependency output file.

-M implies -E, but for illustration '-E' is explicitly specified in this example. This will speed up the make process before first gcc command will stop right after preprocessing is done.

The second time make command is issued, the dependencies files are all available and will participate dependency check on its target. This is why with old software build systems, we often see instructions such as do 'make dep' first, then do 'make' (e.g the pre-2.4 linux kernel build system). The first step generates all the dependencies and the second step does the actual compilation. Of course, we can take out '-E' and '-M' and use '-MD' instead which will produce dependency and object files in a single step. The following is taken from autoconf/make output using 'Makefile' generated from 'configure' command:

/bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I../include -I../include -I/usr/include -g -O2 -Werror -MT binarystring.lo -MD -MP -MF .deps/binarystring.Tpo -c -o binarystring.lo binarystring.cxx

g++ -DHAVE_CONFIG_H -I../include -I../include -I/usr/include -g -O2 -Werror -MT binarystring.lo -MD -MP -MF .deps/binarystring.Tpo -c binarystring.cxx -o binarystring.o

The dependency file foo.d will always depend on the source file foo.c from which it's generated. So whenever we update foo.c and issue 'make', a new dependency file will be generated. Viola, we remember now we were required to do 'make dep' and 'make' with this kind of makefile system.

1. man gcc