Oct 062008

A reader responded to my March article on Parallel Make in Win32 with a question:

I’m trying to get the *.idl files to compile while using the -j flag. I’m having a problem when you run make -j and the idl compiler compiles the *.idl files (that works just fine), but right after that, it tries to run the g++ compiler and if certain files are not created before this command is run, then you get errors saying that there are missing files.

It has been several months since I last worked with make (I left Cooliris in May) and since then most of what I had rediscovered about make has returned to its ‘maker’.  I’ve learned and forgotten make’s obscure rites more times than I care to remember.  There may well be a way to declare that the .h files required by some .cpp files are dependent upon certain .idl files, but the specific details escape me.

However, even if there is not a silver bullet dependency declaration available, you can use a partitioning technique to solve your problem:  Partition your makefile (build process) into stages.  Each stage can run its internal tasks in parallel, but executes sequentially with the other stages.

 Here’s how to do that: 

  1. Set up your makefile so that you have one target (rule) that processes your *.idl files.  Let’s call it IDLCompile
  2. Set up another target that compiles your C++ code, including the C++ code that requires the *.idl output.  Let’s call it CppCompile.
  3. Set up a third target which runs make recursively, passing different target params on each invocation.  I usually make this the default target as well.  Like this: (pseudo code)
    make -j IDLcompile 
    make -j CppCompile
    make etc

You can have all three targets in the same makefile that recursively invokes itself with different arguments. If that gives you a headache you can put the three targets in separate makefiles and invoke make with an additional makefile file argument.  It doesn’t make any difference to make, as each invocation is a separate process anyway.

Now, when you run make with no arguments for a typical build, it will run the steps in the default target sequentially (because you haven’t specified -j).  The stages execute sequentially to each other, but the tasks within each stage can execute in parallel if desired.  You still have parallelism where it counts.

Having make invoke each build stage separately and explicitly gives you simpler, clearer control over what gets built and when than by using traditional file dependency rules. With the topmost make process running in sequential mode, each stage must fully complete before the next stage can begin.  Within each stage, multiple targets can be spun up in parallel, but between stages is a strict serial execution sequence.

It will be very interesting to see how build processes and techniques evolve as distributed computing becomes more and more mainstream thanks to the combination of multicore processors and “elastic computing” in the cloud. Make has served us well for decades across a variety of hardware configurations, including clusters and peer networks, but “clear”, “simple”, and “easy” are never heard in the vicinity of “make”.  There’s room for improvement.