Sunday, February 13, 2011

Using original WWF artwork on your Linux box

Disclaimer: This information is for educational purposes only. If you perform the procedure described here, you assume full responsibility for any and all consequences, including legal action from the copyright holder.

Of course you can create original artwork for .wwf on your Linux box, but wouldn't it be nice if you can use the professional artwork Matt von Jung created for this? But how do you obtain it? Very simple: use the OS/X package. First, download it. Then unzip it:
cd SAVE_AS_WWF_1.0.pkg/Contents

The next step is to decompress the .pax archive:
gzip -d Archive.pax.gz

This seems like a dead end, but it isn't. The .pax archive is simply a cpio file:
cpio -i < Archive.pax
cd 'SAVE AS'

We need the .icns files. That seems like a dead end as well, but not quite. We're almost ready. If you want to extract the icons on your own machine you have to install icns2png, which can be obtained here. Installing it is pretty straight forward:
tar -xzvf libicns-0.7.1.tar.gz
cd libicns-0.7.1
make install

It prefers you install Jasper or OpenJPEG, but I didn't have much luck with that. You don't need it either unless you want to extract the high resolution icons. After you've installed it, enter:
icns2png -x wwf.icns
Reading icns family from wwf.icns...
  Saved 'is32' element to wwf_16x16x32.png.
  Saved 'il32' element to wwf_32x32x32.png.
  Saved 'it32' element to wwf_128x128x32.png.

Done! You can also extract the icons online. It depends on what you prefer, the result is the same.

If you need still another format, use ImageMagick better known to some as convert. Now install the icons in the appropriate directory (that depends on your desktop environment) and you're done. Enjoy!

Monday, February 7, 2011

How to reheat 30 year old spaghetti

A long time ago, when I was still a student, I got my hands on a very fascinating book called "The effects of nuclear weapons" from the Atomic Energy Commission. It was filled with diagrams and tables and enabled you to calculate the effects of a nuclear attack. After five long hours, I got my first results. And it was 4 AM. I knew when I stayed on working and calculating, I was gonna lose a lot of sleep.

However, it was 1981 and at my college we had a highly modern machine: a real PDP-11 with a massive memory of 256 kB and two (floppy) disk drives. Those were the days of the NATO Double-Track Decision, which large parts of the Dutch population opposed. I decided this program was going to be my contribution to that discussion. I called it TEONW (The Effects Of Nuclear Weapons). It is full of cynic puns against the Reagan administration.

I spend nights digitizing the graphs in that book, and coding them in BASIC V10B-02. I had never heard of structured programming and to debug the program I needed a room as long as the listing. If the program said GOTO 5670 I really crawled to line 5670! I never got all the bugs ironed out.

After I had finished college I no longer had access to a computer, but I printed a listing which I entered line by line in the Sinclair ZX Spectrum. I also added some assembly to mimic the green-on-black terminal of the PDP-11.

When the Z80 Spectrum emulator of Gerton Lunter came out, I took the Betadisk, wrote BDDE (BetaDisk Dump Extractor) and transferred the program to a .Z80 file, which may still be found on the Internet. Ten years had gone by.

The .Z80 file was moved from MS-DOS to Linux. I extracted the code with FUSE-utils "listbasic" and made it run under "blassic". By then twenty years had passed.

In the meanwhile, I had been busy writing my 4tH compiler, which is a highly portable bytecode Forth compiler. A few years ago, I added floating point support. What had begun its life as a little toy was now powerful enough to run a program like TEONW. Thirty years later.

TEONW is a relatively small program, just 13 kB source, but it is such an awful mess that I was barely able to understand and code the entry of the basic parameters: yield, altitude and population density. If I ever wanted to convert this program I needed some help. Badly..

TEONW consists of about 500 lines, each with its own line number. Since it is written in Minimal BASIC, every IF-statement is followed by a GOTO. No ELSE, sorry. In order to expose the structure, I needed to get rid of all superfluous labels, i.e. the labels which were not jumped to by either a GOSUB or GOTO. For that I wrote a simple 4tH program, the "uBasic unlabeler" or ubulabel.4th for short. It parses the BASIC program, makes a list of all GOTO and GOSUB labels and then removes all unused labels. That reduced the number of labels to about 100.

But I had still very little insight in the structure of the overall program. Comment was scarce and terse. Instead of making a flow diagram myself, I decided to let the computer do that for me. If you talk about generating diagrams, you talk about Graphviz. This indispensable tool has saved my life more than once - and it would save it once again.

Generating Graphviz code is trivial. I had written Graphviz converters before and I didn't doubt for a moment that I could pull off this one. 4tH excels in parsing text and I didn't even need a full parser here. Just REM, GOSUB and GOTO. Since 4tH also features a conversion program template, all I needed to do was to fill in the blanks. ub2dot.4th was born.

It is basically a very simple 50-lines program. It keeps track of the line it is parsing and when it encounters a GOSUB or GOTO it generates a Graphviz line. Of course, if no GOSUB or GOTO is encountered, it simply executes the next line. That had to be taken into account as well. But this rule has an exception as well: if the last statement is a GOTO or RETURN, the next line will never be reached from that point.

I generated the diagram from the .DOT source, which resulted in this diagram. The black lines are jumps, the red ones are subroutine calls and the blue ones represent normal program flow.

That may not give you much insight at first glance, but when you look carefully, you see some patterns arise. E.g. the code to present the results of the calculation are at the bottom. You can clearly see the different sections for underground explosions and air blasts. At least it helped to chop up the code into manageable chunks.

Since the underground explosions took the least effort, I decided to code that one first. At least it allowed me to set up a basic structure for the program and test it. But first I had a few design decisions to make.

Since this was an all floating point program I decided to use the 4tH Preprocessor (pp4th). Floating point support is coded in high level 4tH, which is not supported by 4tH itself. This meant the code would be rather unreadable. The preprocessor however does offer special floating point facilities, like the entry of floating point numbers without tedious conversions.

4tH offers two floating point libraries. One is rather bulky and fast with a high precision and its own floating point stack, the other is lean and slow and uses the standard data stack. The latter stores the numbers on the stack in standard form, i.e. mantissa and exponent, which is easy to read - if you know what you're looking at. Another added advantage is that it is easier to convert to the dedicated floating point stack version than the other way around.

Forth is a language that is centered around a data stack. It is considered good style to use as few variables as possible. There are non-trivial Forth programs that use only one or two variables or even none at all! However, given the complexity of the task at hand, I decided to use the stack as little as possible and revert to the BASIC variables instead, 25 in all. I never regretted that decision.

Note I had digitized a lot of graphs and these now popped up as clusters of jump instructions, e.g.:
587 IF J2<=-5 THEN GO TO 2007
590 IF J2<=0 THEN GO TO 690
600 IF J2<=5 THEN GO TO 730
605 IF J2<=12.5 THEN GO TO 770
610 IF J2<=25 THEN GO TO 810
630 IF J2<=40 THEN GO TO 840
640 IF J2<=55 THEN GO TO 880
650 IF J2<=62 THEN GO TO 920
660 IF J2<=84 THEN GO TO 960

These were restored to their original table form:
create calc-under
  620 , ' under>62 ,
  550 , ' under>55 ,
  400 , ' under>40 ,
  250 , ' under>25 ,
  125 , ' under>12.5 ,
   50 , ' under>5 ,
    0 , ' under>0 ,
  -50 , ' under>-5 ,

:this calc-under does>
  >r begin fdup r@ @c s>f f% 10e f/ f<=
  while r> cell+ cell+ >r
  fdrop r> cell+ @c execute

The second entry in the table is a pointer to function, in short: a jump table. The greatest challenge however, was to recreate the spaghetti program flow in structured programming, e.g. from this:
1592 IF J2>=-5 THEN GO TO 1594
1593 GO TO 1596
1594 IF J2<=0 THEN GO TO 1980
1595 GO TO 1600
1596 IF -J5/(J1^.4)>200 THEN GO TO 1600
1597 IF M(1)>M(2) THEN GO TO 1980
1599 IF M(2)<200*(J1^(.4)) THEN GO TO 1980
1600 LET S5=INT ((((((M(2)/1000)*(M(2)/1000))*PI)-(S4/J3))*(J3*.75))+S4)

To this:
J2 f@ f% -5e f<
  height f@ fnegate yield f@ f% 0.4e f** f/ f% 200e f>
    S4 f@ density f@ fover fover f/
    1 m f@ f% 1000e f/ fdup f* pi f*
    fswap f- fswap f% 0.75e f* f* f+ ftrunc S5 f!
    1 m f@ 0 m f@ fover f> >r
    yield f@ f% 0.4e f** f% 200e f* f< r> or
    if casualty-corrections then
  J2 f@ f% 0e f>
    S4 f@ density f@ fover fover f/
    1 m f@ f% 1000e f/ fdup f* pi f*
    fswap f- fswap f% 0.75e f* f* f+ ftrunc S5 f!

I must admit, sometimes I was so desperate that I took refuse to some unconventional techniques in order to get an idea what for Petes sake I was trying to do - thirty years ago.

When I first ran it, it failed obviously. The tedious task of debugging was about to begin. I quickly decided that I would need some special debugging aids in order to complete this task, so I wrote a short routine in both BASIC and Forth that allowed me to examine the variables at certain stages of execution:
9000 PRINT "yield=";J1;" J2=";J2;" density=";J3;" J4=";J4
PRINT "height=";J5
PRINT "P1=";P1;" P2=";P2;" 210J=";R1;" 42J=";R2
PRINT "crater-radius=";S1
PRINT "crater-depth=";S2;" crater-rim=";S3;" S4=";S4;" S5=";S5
PRINT "V1=";V1
PRINT "V2=";V2;" V3=";V3;" V4=";V4;" V8=";V8
PRINT "W1=";W1
PRINT "W2=";W2;" W4=";W4;" Z1=";Z1;" Z2=";Z2
PRINT "Z4=";Z4
FOR N=1 TO 5: PRINT "M(";N;")=";M(N);" "; : NEXT N: PRINT: RETURN

Now you understand why I was so happy that I kept with the original variables instead of going for the full Forth conversion! All in all the basic conversion proved to be pretty good. Only one piece of code needed a full rewrite.

Am I finished yet? No, debugging takes a lot of time - but I'm not in a hurry. At some point in time I will have enough confidence to send it into the world. Hopefully, I will have given it another thirty years of useful life. Time, it wouldn't have had if I had left it in this state.

And that would have been a pity, because it is the oldest program of my hand that survived this long. The rest was left decaying on an ancient 8" floppy. Missing in action, presumably dead. But even if they had survived, would I be likely to repeat this exercise? No, probably not.

Reheating spaghetti takes a lot of time. It's better to cook some fresh pasta. ;-)

Thursday, February 3, 2011

Running shellscripts under Windows

Although many programmers look down on shell scripts, they can be an effective way to quickly realize the required functionality in a minimal amount of time. Especially if you want to tie several already available utilities together.

So when I set down to make a FOSS equivalent of the abysmal and closed source WWF drivers, shell scripts seemed the way to go. I already got PDFtk, Ghostscript and the Poppler utils, so all I needed was a bit of glue between them.

The people at the "Save as WWF" Facebook page wanted a GUI, so I had to throw one in. That wasn't much of a problem either, because we got KDialog, Zenity and Dialog, which cover pretty much all the desktop environments around. Starting with a skeleton script I had once posted I was able to release the first version at Freshmeat within a single day. Several releases followed after the first enhancement requests came in. Until that day..

A user asked me whether a Windows port was possible. Wow.. MS-DOS batchfiles are so crippled that converting them was not an option and I am unfamiliar with the newer Windows scripting facilities like PowerShell. Since I use Windows very little I didn't feel like learning that one. So I asked myself the question, can I run the scripts within Windows?

Running them within Windows means you have to fool them into thinking they're running in a Unix environment. Since it seemed pointless to let users install hundreds of megabytes just to run a couple of shellscripts I settled for MSYS. MSYS is just a few megabytes and offers almost all the Unix commands and facilities you're familiar with. Furthermore, it does a pretty good job of silent conversion between the Unix and Windows PATH conventions.

The next question was, can I get Windows ports of all the utilities required. The answer is yes, there are ports of Zenity, WGET, Poppler utilities, Ghostscript and the PDF toolkit. All bases are covered, now let's rock 'n roll!

To my surprise the installation script almost worked out of the box. The only thing was that whoami is not part of MSYS. I know that you use id -un nowadays, but that wasn't much help, because although id correctly returns my Windows login, I used whoami to see if the script was running with root privileges. I decided to make my own whoami in /usr/local/bin, doing a simple echo. That didn't seem like much of a problem, because if the Windows user in question wasn't running with admin privileges, he wouldn't come that far anyway.

The second thing I had to fix was the #!/bin/bash line. MSYS doesn't "have" bash, only sh. Since sh is simply a link to bash, I decided to change it to #!/bin/sh. A few unquoted environment variables posed another problem, but that was quickly fixed. Finally, Ghostscript comes with a whole lot of Unix shellscripts, which I decided to use. Unfortunately, there are two annoying problems:
  1. Ghostscript isn't added to the PATH;
  2. The Ghostscript shellscripts assume gs is installed, which isn't.
The first one was fixed by adding the lib directory of Ghostscript to the PATH and the second was fixed by scanning for the Windows executable and create a small gs shellscript in /usr/local/bin. Then it ran like a charm.. from the prompt!

The Windows version of Zenity is very good, installing it is a breeze. It adds the Zenity executable to the path automatically. But in order to make it run as advertised you have to wrap the whole thing in an MS-DOS batchfile. Since the shellscripts are installed in /usr/local/bin I needed that path in the PATH, which you achieve by adding the --login switch to sh.

I decided to generate all these tiny scripts and batchfiles from the Windows installation script and that was the moment I really stumbled into the annoyance of Windows backslashes. The backslash is Unix' escape character, which makes it particularly difficult to handle. To write a Windows batchfile I decided to use the following sequence:
echo -n "bg_start ..\..\bin\sh.exe --login -c wwf2pdf" > /usr/share/wwftk/bg_wwf2pdf.bat
echo -e "\r" >> /usr/share/wwftk/bg_wwf2pdf.bat

The -n switch suppresses the terminating linefeed, while the -e switch enables the expansion of special control sequences. Which in this case results in the familiar carriage return-linefeed sequence.

The BGstart utility is required to suppress the launching of the sh window, which is featured so prominently in some ported FOSS programs. A few shortcuts and we're done. The icons for the shortcuts were converted online. Quickly and painlessly. And then it ran. Or did it?

The backslashes were back with a vengeance. Zenity returns a Windows path, which is gracefully handled by MSYS, but when it was displayed all backslashes disappeared or made the messagebox look very queer.

I never wanted special Windows versions of the scripts, the idea was that one version runs anywhere. So, how could I transparently handle this one? The answer was one single line:
SHOW=`echo "$3" | sed 's|\\\\|/|g'`

This one converts any embedded backslashes to slashes. Why slashes, you may ask. Well, first it settles the matter once and for all and second, it reminds the Windows user he owes this functionality to a Unix script.

Of course, there are many things that could be improved. A true Windows installer may make it all much easier for the casual user, but again, I'm not prepared to put so much effort into a skill I rarely use. Furthermore, if people think it is important they will step up and solve it. Because this is FOSS.

I'm now looking for someone who does the port to OS/X, so we cover the entire spectrum. If only to prove that the community does a better and faster job than a closed source company..