Using distcc over a heterogeneous environment

Update: New instructions for PowerPC cross compiler and distcc

Update2: New instructions for MIPS cross compiler and distcc

Update3: old version of the instructions for gcc-4.2.2.

Being a kernel hacker, I find that I need to build several kernels a day. Some of the changes I do causes a full recompile of the kernel. If I'm testing a distribution kernel (with most options configured), a full kernel can take up to 40 minutes on some of my best machines. I prefer to compile kernels on the machines I'm testing on to facilitate my reboot scripts. This means that if I'm testing on an older box, those compiles take even longer.

I've been told to try out distcc to ease my pain. I took a look, and I really wanted to try it. The problem I have is that I do not own a homogeneous environment. My machines run Fedora (several versions), Red Hat Enterprise Linux, and even Debian. Not to mention that these boxes are a mix of 64 bit x86_64 and 32 bit i386. Even the 64 bit boxes are sometimes booted into a 32 bit kernel.

Having a heterogeneous environment like this makes distcc a bit of a problem. What distcc does is to have remote boxes compile parts of the build using the remote's compiler. If you have different versions of gcc, this may cause strange bugs to appear if you are lucky (or in this case unlucky) to even get the build to finish a compile.

I use to build cross compilers for things like MIPS, ARM, and PowerPC back in a previous life and I know the pain of building a compiler. So I simply put it off.

Finally, I submitted a bunch of patches for the Linux kernel, and complaints came in that I broke different configurations, and that I should run "make randconfig" which creates random configurations for the kernel. My patches include a bunch of different configurations and I had spent enough time (days) testing each patch, and different configurations that I added. Now I'm asked to also test randconfig. Remember, some of these builds took 40 minutes each.

I finally broke down and said it's about time to get distcc working. The first thing to do is to figure out a way to get a unique version of gcc and binutils across all my platforms and be able to have all build both 32 bit i386 and 64 bit x86_64.

I thought about grabbing some source RPMs and or Debian packages, but distrobutions like to add their own fixes to things like this, and I was still worried about strange bugs that this would cause. I decided to download gcc 4.5.2 and binutils.

Building binutils and GCC across multiple environments

Now the fun begins. Trying to get binutils and gcc working for both 32 bit and 64 bit x86 on 32 bit and 64 bit platforms. One suggestion that was given to me was simply to build a 32 bit gcc that could compile 64 bit and place that on all the boxes. This became a bit of a pain to try to figure out, so I finally gave up. Since 64 bit gcc can easily compile to a 32 bit object, I decided that the 64 bit boxes will simply have a native compiler. The 32 bit boxes will need to hold two compilers:

  1. 32 bit native compiler.
  2. 64 bit cross compiler.

The native 64 bit compiler

Building a compiler for the native environment is pretty trivial. Once you know all the options you need to make. Luckily, distcc does not require anymore than the assembler and compiler. We only need to build binutils and gcc, and (thankfully) not glibc.

[rostedt@bxrhel5 rostedt]$ mkdir dist
[rostedt@bxrhel5 rostedt]$ cd dist
[rostedt@bxrhel5 dist]$ wget
[rostedt@bxrhel5 dist]$ wget
[rostedt@bxrhel5 dist]$ tar xvf binutils-2.20.tar.gz
[rostedt@bxrhel5 dist]$ tar xvf gcc-core-4.5.1.tar.bz2
[rostedt@bxrhel5 dist]$ mkdir binutils-build
[rostedt@bxrhel5 dist]$ cd binutils-build

[rostedt@bxrhel5 binutils-build]$ ../binutils-2.20/configure --prefix=/usr/local/dist \
   --program-prefix=dist- --without-doc --enable-targets=all --enable-languages=c

[rostedt@bxrhel5 binutils-build]$ make -j4
[rostedt@bxrhel5 binutils-build]$ su
[rostedt@bxrhel5 binutils-build]# make install
[rostedt@bxrhel5 binutils-build]# exit
[rostedt@bxrhel5 binutils-build]$ cd ..
[rostedt@bxrhel5 dist]$ mkdir gcc-build
[rostedt@bxrhel5 dist]$ cd gcc-build

[rostedt@bxrhel5 gcc-build]$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc CC=gcc AR=dist-ar \
   AS=dist-as LD=dist-ld NM=dist-nm ../gcc-4.5.1/configure --prefix=/usr/local/dist \
   --program-prefix=dist-  --without-doc --enable-bootstrap

[rostedt@bxrhel5 gcc-build]$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc CC=gcc AR=dist-ar \
   AS=dist-as LD=dist-ld NM=dist-nm make -j4

[rostedt@bxrhel5 gcc-build]$ su
[rostedt@bxrhel5 gcc-build]# make install

There, that's all! Of course this still isn't enough to use distcc, but it is a major part. Let's now look at what to do for the more complex i386 boxes.

The native 32 bit compiler

This is almost as easy as the 64 bit compiler, except that it has a slight bug. After doing the configure (I saw this on Fedora, Red Hat Enterprise Linux and Debian i386 boxes), the makeinfo is considered missing. If you actually have makeinfo, then you can just fix the Makefile and continue.

rostedt@goliath:~/dist/binutils-build$ which makeinfo

rostedt@goliath:~/dist/binutils-build$ vim Makefile
[in vim] /missing

You should see something similar to this:

BISON = bison
YACC = bison -y
FLEX = flex
LEX = flex
M4 = m4
MAKEINFO = /home/rostedt/dist/binutils-2.20/missing makeinfo
EXPECT = expect
RUNTEST = runtest

Simply remove the "missing" part of the line and save the file.

BISON = bison
YACC = bison -y
FLEX = flex
LEX = flex
M4 = m4
MAKEINFO = makeinfo
EXPECT = expect
RUNTEST = runtest

NOTE: With binutils-2.20 I did not need to do the above.

Continuing on with the building.

rostedt@goliath:~$ mkdir dist
rostedt@goliath:~$ cd dist
rostedt@goliath:~/dist$ wget
rostedt@goliath:~/dist$ wget
rostedt@goliath:~/dist$ tar xvf binutils-2.20.tar.gz
rostedt@goliath:~/dist$ tar xvf gcc-core-4.5.1.tar.bz2
rostedt@goliath:~/dist$ mkdir binutils-build
rostedt@goliath:~/dist$ cd binutils-build

rostedt@goliath:~/dist/binutils-build$ ../binutils-2.20/configure --prefix=/usr/local/dist \
  --program-prefix=dist- --without-doc --enable-64-bit-bfd

rostedt@goliath:~/dist/binutils-build$ vim Makefile

  # This is where I remove the "missing" part of makeinfo (but not for 2.20 though)

rostedt@goliath:~/dist/binutils-build$ make -j4
rostedt@goliath:~/dist/binutils-build$ echo $?
rostedt@goliath:~/dist/binutils-build$ su
rostedt@goliath:/home/rostedt/dist/binutils-build# make install
rostedt@goliath:/home/rostedt/dist/binutils-build# echo $?
rostedt@goliath:/home/rostedt/dist/binutils-build# exit
rostedt@goliath:~/dist/binutils-build$ cd ..
rostedt@goliath:~/dist$ mkdir gcc-build
rostedt@goliath:~/dist$ cd gcc-build

rostedt@goliath:~/dist/gcc-build$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc CC=gcc \
   AR=dist-ar AS=dist-as LD=dist-ld NM=dist-nm ../gcc-4.5.1/configure --prefix=/usr/local/dist \
   --program-prefix=dist-  --without-doc --enable-bootstrap --enable-targets=all

rostedt@goliath:~/dist/gcc-build$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc CC=gcc \
   AR=dist-ar AS=dist-as LD=dist-ld NM=dist-nm make -j4

rostedt@goliath:~/dist/gcc-build$ echo $?
rostedt@goliath:~/dist/gcc-build$ su
rostedt@goliath:/home/rostedt/dist/gcc-build# make install
rostedt@goliath:/home/rostedt/dist/gcc-build# echo $?

Not too painful at all!

The 64 bit cross-compiler on 32 bit boxes

Compiling a cross compiler takes a little more work. This may not be the best way, but it worked for me. I install the cross-compiler in the same location as the native distcc compilers, but I give them a different program prefix.

NOTE: The following has a different program prefix.

We save ourselves the trouble of downloading the tools again, since they are already in the directory root that we will compile. We only need to create a new build directory.

rostedt@goliath:~/dist$ mkdir binutils-build-x86_64
rostedt@goliath:~/dist$ cd binutils-build-x86_64

rostedt@goliath:~/dist/binutils-build-x86_64$ ../binutils-2.20/configure \
    --prefix=/usr/local/dist --program-prefix=dist-x86_64-  --without-doc \

rostedt@goliath:~/dist/binutils-build-x86_64$ vim Makefile

  # once again fix the "missing" line (not needed for 2.20)

rostedt@goliath:~/dist/binutils-build-x86_64$ make -j4
rostedt@goliath:~/dist/binutils-build-x86_64$ echo $?
rostedt@goliath:~/dist/binutils-build-x86_64$ su
rostedt@goliath:/home/rostedt/dist/binutils-build-x86_64# make install
rostedt@goliath:/home/rostedt/dist/binutils-build-x86_64# echo $?
rostedt@goliath:/home/rostedt/dist/binutils-build-x86_64# exit
rostedt@goliath:~/dist/binutils-build-x86_64$ cd ..
rostedt@goliath:~/dist$ mkdir gcc-build-x86_64
rostedt@goliath:~/dist$ cd gcc-build-x86_64

Once again, note the program prefix difference between the 64 bit and 32 bit.

rostedt@goliath:~/dist/gcc-build-x86_64$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc \
   AR=dist-x86_64-ar AS=dist-x86_64-as LD=dist-x86_64-ld NM=dist-x86_64-nm  \
   ../gcc-4.5.1/configure --prefix=/usr/local/dist --program-prefix=dist-x86_64-  \
   --without-doc --enable-bootstrap --target=x86_64-unknown-linux  \
   --disable-threads --enable-shared --disable-decimal-float

UPDATE: Found that -fstack-protect-all set will cause the cross compiler to create "stack_chk_guard" calls. These are not defined in the kernel and cause linking problems.

To solve this, after doing the configure, you need to hack the gcc/autohost.h file. Find the TARGET_LIBC_PROVIDES_SSP and change it from the uncommented out undef, to a define.

/* Define if your target C library provides stack protector support */

change to:

/* Define if your target C library provides stack protector support */

Now continue with the make.

rostedt@goliath:~/dist/gcc-build-x86_64$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc \
   AR=dist-x86_64-ar AS=dist-x86_64-as LD=dist-x86_64-ld NM=dist-x86_64-nm make -j4

rostedt@goliath:~/dist/gcc-build-x86_64$ echo $?

WHAT!!!! The compile failed??

NOTE: For gcc-4.5.1 it did not fail. Just go on to su; make install.

Don't panic! We can ignore this. Well, not really ignore it, but at least work around it. I'm sure there's something that I don't understand, and probably missing a flag, but this work-around worked, so that's what I did.

Now edit the file from the build directory

, and search for crti.o. You should see something like:

%{static|static-libgcc:-lgcc -lgcc_eh}%{!static:%{!static-libgcc:%{!shared-libgcc:-lgcc --as-needed -lgcc_s --no-as-needed}%{shared-libgcc:-lgcc_s%{!shared: -lgcc}}}}

%{!shared: %{pg|p|profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s}}    crti.o%s %{static:crtbeginT.o%s;shared|pie:crtbeginS.o%s;:crtbegin.o%s}



Remove the crti.o%s

%{static|static-libgcc:-lgcc -lgcc_eh}%{!static:%{!static-libgcc:%{!shared-libgcc:-lgcc --as-needed -lgcc_s --no-as-needed}%{shared-libgcc:-lgcc_s%{!shared: -lgcc}}}}

%{!shared: %{pg|p|profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s}}    %{static:crtbeginT.o%s;shared|pie:crtbeginS.o%s;:crtbegin.o%s}



Save it and continue with the compile.

rostedt@goliath:~/dist/gcc-build-x86_64$ vim gcc/specs

  # make the fixes as described above

rostedt@goliath:~/dist/gcc-build-x86_64$ PATH=/usr/local/dist/bin/:$PATH:`pwd`/gcc \
   AR=dist-x86_64-ar AS=dist-x86_64-as LD=dist-x86_64-ld NM=dist-x86_64-nm make -j4

rostedt@goliath:~/dist/gcc-build-x86_64$ echo $?

WHAT! It failed again??

Yes, it will fail shortly afterward complaining about not having libc or something. We can simply ignore it. We have the compiler that we want. Now it's time to install it. This is a bit more manual than before.

rostedt@goliath:~/dist/gcc-build-x86_64$ su
rostedt@goliath:/home/rostedt/dist/gcc-build-x86_64# cd gcc
rostedt@goliath:/home/rostedt/dist/gcc-build-x86_64/gcc# cp libgcc*.a /usr/local/dist/x86_64-unknown-linux/lib/
rostedt@goliath:/home/rostedt/dist/gcc-build-x86_64/gcc# make -k install
rostedt@goliath:/home/rostedt/dist/gcc-build-x86_64/gcc# echo $?

That's it for building and installing the cross-compiler. The "-k" switch in the make is to keep it from stopping on the first error it hits. As you see, the install did fail. But we don't care. We have enough installed to continue to distcc.

Setting up distcc for our new compiler environment

Now it is time to setup distcc to be use the proper compiler and assembler for building kernels and such. To ease this part I've created two tar balls that contain helper files to facilitate using distcc. One is for 32 bit machines and the other is for 64 bit.

These tar balls include the following:

Now I just install this tar ball for the approrpiate box (goliath is i386) and start up distcc (note: I don't go into how to install distcc. That can be either installed by the distro packaging or built manually).

rostedt@goliath:~/dist$ wget
rostedt@goliath:~/dist$ su
rostedt@goliath:/home/rostedt/dist# cd /
rostedt@goliath:/# tar xvf ~rostedt/dist/dist-cc-files-i386.tar.bz2

NOTE: if you changed the location of the compiler or the prefix name, you will need to modify the files in the tar ball to accomodate.

Once you have distcc installed and the servers running as well as your DISTCC_HOSTS environment variable setup. All you need to do to compile something like the kernel is this:

rostedt@goliath:~/linux-2.6.24$ distmake-32 -j50

or for a 64bit kernel:

[rostedt@bxrhel5 linux-2.6.24]$ distmake-64 -j50

That's it!

See also

My setup isn't that dynamic, so I can get by with just updating my DISTCC_HOSTS when I need to. But incase you want dynamic lookups of available distcc servers, I recommend looking into this patch.

A tar ball of my i386 set up (untar in /).

A tar ball of my x86_64 set up (untar in /).

Good luck, and happy compiling!