Sunday, December 30, 2007

Cross compiler blues

It is the dilemma of every single FOSS developer. Sure, you want as many people as possible to use your software but you also want to acquaint them with the advantages of FOSS as well. Some developers think it is a non-question: they don't give portability a single thought. Others do, but think it is better to force users by not making their program available under other Operating Systems. "If you want to use my program, that's cool, but switch to Linux first". Other developers, like me, think that it is better to let people use FOSS software under their current Operating System because nobody will make the switch for a single program. It is the strategy that projects like Firefox are using too. It is based on the presumption that when the reasons for switching are piling up the transition is less painful because they can continue to use the programs they know and love.

But that puts us developers into an another painful dilemma. In order to provide these packages we have to have access to these platforms. And that is not something we do lightly. First of all, there is the cost. Commercial Operating Systems are expensive - and let's face it - a pain in the neck due to all the copy protection features. Apart from that, why should you burden yourself with a more complex configuration, another development system and a new learning curve?

Well, there is a solution but it has it's drawbacks too: cross compilation. That means you can continue to use your current Operating System and development system but still are able to support other platforms. It seems too good to be true and - frankly - it is. Almost no distribution has cross development packages in its repository and I think that is a shame. Sure, you can try to compile them yourself but - believe me - that ain't that easy. I have been considering cross compilation for a long time because I didn't want to turn to a MS-Windows machine each and every time I wanted to make a new release. It is really a question of dependency because I always have to rely on my employer to provide such a beast. I run Linux at home and nothing else.

The procedure was always the same. Convert the sources to Microsoft text, put them on a memory stick, take the memory stick wherever the MS-Windows machine was located, copy the sources to the system, compile them, test the compilants, make the package, copy the package back on the memory stick, take the stick home, mount the stick, copy the package to the proper location and we're done. It was even worse when I switched employers because I had to rebuild the entire development system. It usually doesn't come on a vanilla MS-Windows machine nowadays, you know. Who's gonna use a C compiler when you're an IT consultant? Don't bug me with comments like doing this stuff during working hours. We got lunchtime and there is also overtime. It doesn't take me hours to make a package of a simple project, it's more like minutes.

But like I said, cross compilation isn't easy as well. A lot of information is outdated and when not it is mindboggingly complicated. I'm not afraid to compile KOffice or other 50 meg source packages - as a matter of fact that is the standard routine here since I'm still using SUSE 9.2. Never change a running system - but that is not important right now. Another consideration is that just because cross compilation is so complicated I'm afraid to break my current development system. Two compilers on the same system: is that gonna work?

I can tell you that it works. It works even very well. But you have to know where to start and how to get started. A good start is a simple script by Volker Grabsch that builds an MS-Windows C cross compiler and comes with a host of libraries. Just download it - I assume you know how to unpack a .tar.gz - and make a few simple preparations. First of all you have to decide where your compiler is going to reside. I chose /opt/mingw but any other location is alright too. Just remember to edit your .bashrc script in order to change $PATH:
export PATH=/opt/mingw/bin:$PATH

Next, create the directory. If you're afraid that the script might touch your current development environment, be sure you give yourself full access because you can run the entire script under your current user account:
cd /opt
mkdir mingw
chown habe mingw
chgrp users mingw

Be sure to use the user account settings appropriate for your system; this is just an example. Next, make a tiny adjustment to the script. Don't take another route, you won't get a proper compilant. Line 96 to be exact:

Now run it. Take a coffee and wait. Make it a big one. If all goes well you'll end up with a cross compiler in /opt/mingw and you're almost ready to run. You probably will have to make a few adjustments to your Makefile:

You may have to add a few others, depending on your project. Please refer to this page on cross compilation or the comments on the freshmeat page. All you have to do is type this:
make CROSS="i386-mingw32msvc-"

Depending on your project this may be a bit more complex, but now you got plenty of links to solve these issues. If you're using configure, all you have to do is:
./configure --host="i386-mingw32msvc"

or this:
./configure --target="i386-mingw32msvc"

Whatever works for you. In my case, it worked fine and I was very happy. So I got reckless and thought I could do the same for MS-DOS. I did some research and found the proper packages. Seemed easy enough, so I downloaded the binutils, crx and gcc packages, became root and installed them with:
rpm -Uvh package.rpm

Not much happened. I usually don't use source RPMs since I prefer .tar.gz. I faintly remembered that the last time I used them the beast built a binary package that I could install as usual. But that was several years ago. It could be a lapse of memory, I'm getting older too. After some time I found that they had wound up at /usr/src/packages and that was it. Note this location can be different on your system, you will have to find out for yourself. I decided to build them:
rpmbuild -bb djcrx.spec
rpmbuild -bb djcross-binutils.spec

It took another coffee, but there was nothing wrong with that. I ended up with two beautiful packages in RPMS:
rpm -Uvh djcrx-2.04pre-5.noarch.rpm
rpm -Uvh djcross-binutils-2.17-5.i686.rpm

Okay, next one:
rpmbuild -bb djcross-gcc.spec

Several requirements were not met. I had to have GNAT (what the hell is that??) and older versions of autoconf and automake. Now we're getting in dangerous territory. Two different versions of such crucial tools? Are they completely insane?! I found GNAT. That is an Ada compiler. Well, I can always remove them later. I installed those from DVD. Then the next hurdle, autoconf. I got that one from the GNU repository. First, I check where my autoconf was:
which autoconf

Then I built the old autoconf as a regular user:

Finally I figured out where the beast would install itself:
make -n install

That proved to be /usr/local/bin. Fair enough. So I ran make install as superuser and renamed all executables, e.g.:
cd /usr/local/bin
mv autoconf autoconf-2.13

So, I had managed to resolve most - if not all - dependencies. I just had to edit the specfile (the horror):
sed -e 's:^\(AUTOCONF_OLD=\).*$:\1/usr/local/bin/autoconf-2.13:' \
-e 's:^\(AUTOHEADER_OLD=\).*$:\1/usr/local/bin/autoheader-2.13:' \

Now try again:
rpmbuild -bb --nodeps djcross-gcc.spec

And yes, it ran, it ran!! I took a coffee. I took another one. My girlfiend was sleeping on the bench by now. I made another pot of coffee. Hours were ticking by. It seemed as if the beast was building and rebuilding itself over and over again. The city was completely silent by now. And then the ultimate disaster: the Ada compiler build broke off with an error. Obviously, the build tried to create a whole bunch of compilers, not only C. For my own projects, I only use C. I do not create much F77, Ada or C++ packages for myself or others - zero to be exact - nor do I have any intention to do so. I went to study the spec file again. It seemed you could switch off several compiler builds, which is alright with me. I only want C. So I changed these lines:
--enable-languages=c,ada \
--enable-languages=c,c++,f95,objc,obj-c++,ada || exit 1

to this:
--enable-languages=c \
--enable-languages=c || exit 1

No mercy! Then I prepared another pot of coffee, sighed and tried again. Several hours later, the beast was done. Victory! I've done it, I've done it! It may or may not run, but I have completed the build! Out came a tiny 7 meg RPM.. Now install:
rpm -Uvh djcross-gcc-4.2.2-12ap.i686.rpm

I tested it and it worked fine:
make CROSS="i586-pc-msdosdjgpp-"

It installs itself into /usr/bin, but now I know that is not a problem. I was appalled by the way I had to make this work. It is these kind of builds that gives Linux a bad rap. And why? It is possible to change an old autoconf to new ones. It is easy to add a simple README or make a simple webpage concerning cross compilation that is not outdated. There is a lesson to be learned here, especially for those who maintain these cross compilation packages. Don't take the easy way, make a little effort. It does not only help FOSS, but also benefits your prospect users. If you don't take their way, they take the highway.

At least, for me the work is done. I'm very happy with my uptodate cross compilers since I can do all my compilation and packaging in the comfort of my Linux box. That's worth a night of hacking and a annoyed girlfriend, isn't it. Well, where the girlfriend is concerned, I'm not too sure.. ;-)


Unknown said...

I've too started doing cross compilations and it cuts down release times enormously as we provide an executable for Windows users. On Ubuntu it is really easy to get going, just install the various bits (from the universe repository):

sudo apt-get install mingw32 mingw32-binutils mingw32-runtime wine

Then build your package adding the following options to your configure step:

./configure --host=i586-mingw32msvc --build=i686-linux

The installation step above includes wine, so once you have compiled your package you should even be able to run the resulting Windows executable on Linux and possibly your entire set of regression tests if you have them. All without the headaches of configuring and maintaining a Windows build machine :)

Anonymous said...

Before I developed mingw_cross_env, I used the mingw32 Debian package, too. However, this solves only the simple part: Creating a cross compiler.

However, you also need to port the libraries used by your application, and many libraries need special tweaks beyond "--host=...". This is the essential problem! Building a cross compiler is comparatively trivial.

I started with repackaging the Gnuwin32 packages. However, Gnuwin32 has no transparent build process, and the binary packages are not as uniform as e.g. RPM or Debian packages. I had no insight in how the Gnuwin32 authors built their packages, and I needed many libraries they didn't offer.

The next experiment was porting some Debian packages using the well-patched Debian sources and dpkg-cross. However, for some libraries there is too much to do for win32, so I worked more against the Debian tools than with them.

It became clear that I need something similar to the BSD ports collection or the Gentoo portage system. So I created a simple but comprehensive shell script which built every library from source.

First, it cross compiled only the libraries. Then I took the opportunity to build the cross compiler there as well, making it independent from Debian. Feel free to remove the code sections that create binutils and gcc, and to use the mingw32 package instead. But this won't simplify much, because the essential problem remains porting the libraries.

Anonymous said...

You know.
With debian, it is much more easy to install a cross compiler; it has precompiled libs:

$ apt-get -Y install mingw32
$ i586-mingw32msvc-cc hello.c
$ wine a.exe

Anonymous said...

Any word on how to make a cross-compiler for macs?

I prefer virtualbox + mingw to ming32 just on linux, but virtualization is beaten off by apple's eula.