Autobuild FAQ

Questions

• 1.1 Why cook?
• 1.2 How do I add a new project?
• 1.3 How do I add third party C/C++ libraries?
• 1.4 How can I accelerate build turnaround?
• 1.5 Can I run multiple concurrent "b"s on the same box?
• 1.6 How do I troubleshoot compile problems?
• 1.7 How do I troubleshoot "cook undefined variable" problems?
• 1.8 How can I exclude portions of the source tree from "b"?
• 1.9 How does the environment affect the build?
• 1.10 What are those [...] variable references used all over the place?
• 1.11 How do I include generated header files?

Answers

1.1 Why cook?

Short answer: cook is a better make. It has crucial features allowing the implementation of a reliable incremental autodiscovering build system.

Long answer: GNU make is a tool which shoehorned many useful features into bizarre constructs, mainly for maintaining backwards compatibility. The bizarre constructs render Makefiles hard to read and maintain.

The following features of cook, not available in GNU make, enable the construction of a reliable incremental parallel autodiscovering build system:

• The #include [cook xxxx] construct, forcing include files to be built prior to inclusion, as opposed to the GNU make paradigm, which includes the existing, out of date version and then rebuilds the include files. The existing include files may affect and render useless the constructed include files.

• The cascade dependency construct, allowing the definition of dependencies via an "if you need this, you also need that" expression.

• Powerful wild card pattern rules.

• Different selection algorithm for rules. In GNU Make, a pattern rule is considered valid if both the target and the dependency pattern match - in other words both sides are evaluated at the same time. In cook, the target side is evaluated first, then the wildcard match is used to evaluate the dependency side. This allows constructs of the form: lib%.a: [objects_for_lib%.a], where one can use the wildcard match to construct a variable name to resolve. There is no direct way to produce the equivalent construct in GNU make - instead, one would need to generate the dependency list separately as an include file, which then falls prey to the first point above.

• Fine grained control over activation and threading of rules.

1.2 How do I add a new project?

1. Find a location in the source tree, create a directory.

2. Decide what gets built in that directory:

   • a set of small programs (simplex)
   • one large program (complex)
   • a static library (archive)
   • an ordinary shared object (dumb_so)
   • a dynamically loadable shared object (smart_so)
   • a set of pre-compiled perl scripts (perl)
   • a set of validated xml files (xml)
   • a set of preprocessed shell scripts (sh)
   • a set of preprocessed NT batch files (bat)
   • rpms (rpm)
   • a maven project (maven)
   • something using its own build system (sub_build)
   • a one-off build via cook (custom)

3. Add a Howto.cook file with at minimum one line containing:

           make something;
   

   where something is one of the things listed above. For example:

           make complex;
   

4. Add additional customizations as needed (see Tutorial).

5. Add code.

6. Run b.

1.3 How do I add third party C/C++ libraries?

Assuming you have the source tree laid out as described in the tutorial, proceed as follows:

If you have a binary distribution:

1. Pack the binary distro into a tarball. Ensure that the tarball unpacks into a directory with the same basename. In other words, ensure that foo-2.7.tar.gz unpacks into foo-2.7.

2. Check the tarball into the 3party/platform directory.

3. Go to cook-install-prefixshare/cook/autobuild/include/use and clone an existing third party product, for example jdk1.5.0_09.

   If foo-2.7 is a C++ library with some header files, it is likely to have the following structure:

       foo-2.7/
          |_include/
          |   |_foo.h
          |_lib/
              |_libfoo.a
   

   The use file for foo-2.7 will have at least the following content:

   /* simplify the expressions to follow - not used elsewhere */
   foo-2.7_dir = [thirdparty]/[variant]/[unix]/foo-2.7;
   
   /* cause retrieval and unpacking of tarball */
   foo-2.7_use_unpack = [foo-2.7_dir]/[dummy];
   
   /* cause rebuilds for users of this file, if it changes */
   foo-2.7_use_dependency = ADD [__FILE__]':'[__LINE__] [__FILE__];
   
   /* compiler flags */
   foo-2.7_cc_I_flags = ADD [__FILE__]':'[__LINE__] -I[foo-2.7_dir]/include;
   foo-2.7_cpp_I_flags = ADD [__FILE__]':'[__LINE__] -I[foo-2.7_dir]/include;
   
   /* link dependency */
   foo-2.7_external_lib = ADD [__FILE__]':'[__LINE__] [foo-2.7_dir]/lib/libfoo.a;
   
   /* the ADD [__FILE__]':'[__LINE__] documents
    * the location of the definitions
    */
   

4. Edit the Howto.cook files of the projects using foo-2.7, adding a line:

         use foo-2.7;
   

5. Run b and see if it works.

If you have a source distribution only, you have two options:

• Build it by hand on all platforms, create a binary tarball, then follow the process described above. Use this method if you expect no customizations and no additions.
• Integrate the source distribution into our source tree. Do this if you need to customize the product at the source level.

The best procedure for accomplishing the latter is to create a separate vendor branch for the source distro, then merge the vendor branch into your development tree.

The idea is to have a ready-made location to merge upgrades from the vendor with your own. If the vendor chooses to reorg their source tree, you are kind of hosed, but you may still have a shot at merging it if the reorg is small. Most source trees change rarely, so it's still a good process.

1.4 How can I accelerate build turnaround?

You can use b -par=n to increase the number of build threads. Note that not all portions of the build are completely parallelizable, but most are.

You can run b in a subdirectory, thereby only building the targets therein and their prerequisits.

You can run b skip in a subdirectory, thereby skipping any prerequisits outside the subdirectory. Note that this requires to have done a complete build before, and you need to be sure no dependencies have changed.

Every build step creates a convenience symlink called last_command to the last build script executed in any directory. If you encounter compiler errors in a directory, it may be quickest to just rerun ./last_command until the compilation succeeds.

1.5 Can I run multiple concurrent builds on the same box?

Short answer: No.

Long answer: There is resource contention:

maven

local repo is a shared resource

rpm

transaction lock file is a shared resource

b

the error collection files and the build log files could be clobbered.

1.6 How do I troubleshoot compile problems?

Every build step in b generates a target file and the following additional files:

target.need

Direct prerequisites

target.why

Prerequisites which were out of date

target.how.sh

Shell script to generate target.

target.errors

Output produced by running target.how.sh.

In addition, every failure generates an entry in the file DO/Errors-, which lists all the targets which failed to build.

Running b will also generate a file named Howto.list, which is a copy of the screen output. Use this to examine errors which scrolled off the screen.

The b output takes the following format:

    Operation: sources -> (x)
           (x) -> target

The first line is displayed when construction of the target begins, the second line is displayed when the target is done. The (x) denotes the thread id, thereby permitting correct interpretation of the interleaved output of a parallel build.

Build errors are displayed by first dumping the contents of the target.errors file, then the following output:

### (x) -> Errors detected: target.errors
### (x) -> Command run:     target.run.sh

The paths are always relative to the top of the source tree. You can easily reproduce the error by executing target.run.sh. You can also re-examine the error messages by inspecting target.errors.

The build script target.run.sh will have comments indicating the origin of various settings used in the build.

1.7 How do I troubleshoot "cook undefined variable" problems?

If the undefined variable contains a path, it is usually an indication that someone either introduced an error into the Howto.cook file in the location implied by the path, or that it is a new Howto.cook file which someone forgot to add to version control.

The way b stores localized settings is by using variables named with the path to the directory where that setting applies. The end of the path will often have the variant (debug or optimized) appended to it. This last entry should be ignored when determining the location

1.8 How can I exclude portions of the source tree from b?

If you wish to totally exclude a portion of the source tree, place a file named .do_not_cook in the directory to be excluded.

If you wish to exclude a portion of the source tree for specific variants or platforms, use the exclude-platform list; or the exclude-variant list; constructs.

1.9 How does the environment affect the build?

Setting any of the following environment variables may cause build problems:

 JAVA_HOME
 CDPATH
 CLASSPATH
 ANT_HOME
 JAVA5_HOME
 LITMUS_HOME
 JWSDP_HOME
 JBOSS_HOME
 PLATFORM
 BLDOPT
 VARIANT

Most of these are listed in the Howto.cook file at the top of the source tree and will cause b to abort. This list is unfortunately not complete due to the vagaries of various sub-build handoffs.

1.10 What are those [...] variable references used all over the place?

Cook's [...] construct is the equivalent of GNU make's $(...) construct. b will define a a variety of these variables which can be used to express dependencies, compiler options and commands and other things.

Cook's [...] is also used to invoke functions, again somewhat similar to the built-in functions provided by GNU make. Cook makes it very simple to add new functions, making it look somewhat like lisp, whereas GNU make's $(call ...) semantics are rather bizarre.

In b, variables are scoped by directory. This is accomplished by making the relative path in the branched source tree part of the name. When applicable, the variant (debug, optimized, ...) is appended to that path.

The following is an incomplete list of commonly used variables.

[defined-or-warn build_target_for_path/variant]

[defined-or-warn build_target_for_path/variant]

1.11 How do I include generated header files?

You check in a "docking header file", like this:

  #ifndef _generated_header_h_
  #define _generated_header_h_

  /* The following macro "stringifies" a constant. This allows constants to be
   * passed in via compiler flags without using fancy quoting mechanisms.
   */

  /* Go ask the ANSI committee why you need two levels of indirection here */
  #ifndef cook_str
  #define cook_str(s)  # s
  #define cook_string(s) cook_str(s)
  #endif

  /* Now include the generated header file */
  #include cook_string(OBJDIR/the_generated_header.h)

  #endif

The code above works when the compiler is invoked with the -DOBJDIR=DO/platform/variant. This setting is defined as a default setting for C/C++ compilers.

This in cause the generated header file to be included from the DO/platform/variant directory.

One unfortunate side effect of this strategy is that any other #define which happens to match platform or variant (or DO for that matter) will be substituted. This may be hard to debug.