Makefile improvements: rule consolidation, dependency generation

[longjon]

This PR addresses #599 using the -MMD option of gcc/clang (and the -M option of nvcc) to automatically generate header file dependencies, finally making the build truly incremental.

In addition, some consolidation of build rules is included (more might be possible), and the explicit enumeration of build directories is made a bit more automatic (further improvements in this direction are also possible).

This is pretty fresh so you might want to let it age a couple days in case I messed something up; I'll be working off this branch.

[jeffdonahue]

How does make know the object files also depend on whichever automatically generated *.hpp header dependencies gcc says are needed?

[longjon]

Make merges multiple rules into one with a union over prerequisites, see https://www.gnu.org/software/make/manual/html_node/Multiple-Rules.html. The additional rules are provided in the .d files included in the last line of the Makefile. (Note that on first build, the .d files don't exist, but this doesn't matter, because everything has to be built anyway.)

[jeffdonahue]

Cool, LGTM! Thanks for setting up the automatic dependency generation, that should speed up builds quite a bit.

[sguada]

Nice job

On Wednesday, November 26, 2014, Jeff Donahue notifications@github.com
wrote:

Cool, LGTM! Thanks for setting up the automatic dependency generation,
that should speed up builds quite a bit.

—
Reply to this email directly or view it on GitHub
#1472 (comment).

Sergio

[shelhamer]

Nice! Sorry if I'm blind, but is the nvcc -M part missing?

[longjon]

@shelhamer, that's weird, where did that go? I'm glad somebody is paying attention (and git reflog -G exists).

[longjon]

Two bugs have been fixed.

1. Dependency generation was broken for .cu files.

   nvcc uses a different interfaces from gcc for specifying the name of the target and the output dependency file. In particular, the -odir option needs to be used if output is in a different directory, and nvcc assumes the target will have a .o extension. Rather than hack around that assumption, I changed the build targets for .cu files from .cuo files to .o files that live in a separate tree under build/cuda. This might be a little less convenient for certain purposes, but it's more harmonious with the assumptions of nvcc, and it's unusual to dig for object files in the build directory anyway.

2. make clean was not removing the .build_* directories, and consequently the build symlink was not being recreated after a make clean.

[longjon]

Two more bugs have been fixed.

1. Dependency generation of test object files was broken, because test dependency files were not included.
2. There is an untracked dependency of the generated caffe_pretty_print.pb.cc on the generated caffe.pb.h. I normally did not experience a problem when using make -j, as the race was always won; however, make (-j1) revealed the issue. The fix is to use the old trick for protoc generated files: all .pb.o files depend on all .pb.h ones (there are just two of each).

I know of no other issues that have come up in my use of this branch, so (with @jeffdonahue's approval a few bugs ago) I think I'll merge this now (and #1561).

Note that building may still be slow sometimes, because of the monolithic header files and test binary. I'll make an issue about that shortly.