Thursday, February 3, 2011

Running shellscripts under Windows

Although many programmers look down on shell scripts, they can be an effective way to quickly realize the required functionality in a minimal amount of time. Especially if you want to tie several already available utilities together.

So when I set down to make a FOSS equivalent of the abysmal and closed source WWF drivers, shell scripts seemed the way to go. I already got PDFtk, Ghostscript and the Poppler utils, so all I needed was a bit of glue between them.

The people at the "Save as WWF" Facebook page wanted a GUI, so I had to throw one in. That wasn't much of a problem either, because we got KDialog, Zenity and Dialog, which cover pretty much all the desktop environments around. Starting with a skeleton script I had once posted I was able to release the first version at Freshmeat within a single day. Several releases followed after the first enhancement requests came in. Until that day..

A user asked me whether a Windows port was possible. Wow.. MS-DOS batchfiles are so crippled that converting them was not an option and I am unfamiliar with the newer Windows scripting facilities like PowerShell. Since I use Windows very little I didn't feel like learning that one. So I asked myself the question, can I run the scripts within Windows?

Running them within Windows means you have to fool them into thinking they're running in a Unix environment. Since it seemed pointless to let users install hundreds of megabytes just to run a couple of shellscripts I settled for MSYS. MSYS is just a few megabytes and offers almost all the Unix commands and facilities you're familiar with. Furthermore, it does a pretty good job of silent conversion between the Unix and Windows PATH conventions.

The next question was, can I get Windows ports of all the utilities required. The answer is yes, there are ports of Zenity, WGET, Poppler utilities, Ghostscript and the PDF toolkit. All bases are covered, now let's rock 'n roll!

To my surprise the installation script almost worked out of the box. The only thing was that whoami is not part of MSYS. I know that you use id -un nowadays, but that wasn't much help, because although id correctly returns my Windows login, I used whoami to see if the script was running with root privileges. I decided to make my own whoami in /usr/local/bin, doing a simple echo. That didn't seem like much of a problem, because if the Windows user in question wasn't running with admin privileges, he wouldn't come that far anyway.

The second thing I had to fix was the #!/bin/bash line. MSYS doesn't "have" bash, only sh. Since sh is simply a link to bash, I decided to change it to #!/bin/sh. A few unquoted environment variables posed another problem, but that was quickly fixed. Finally, Ghostscript comes with a whole lot of Unix shellscripts, which I decided to use. Unfortunately, there are two annoying problems:
  1. Ghostscript isn't added to the PATH;
  2. The Ghostscript shellscripts assume gs is installed, which isn't.
The first one was fixed by adding the lib directory of Ghostscript to the PATH and the second was fixed by scanning for the Windows executable and create a small gs shellscript in /usr/local/bin. Then it ran like a charm.. from the prompt!

The Windows version of Zenity is very good, installing it is a breeze. It adds the Zenity executable to the path automatically. But in order to make it run as advertised you have to wrap the whole thing in an MS-DOS batchfile. Since the shellscripts are installed in /usr/local/bin I needed that path in the PATH, which you achieve by adding the --login switch to sh.

I decided to generate all these tiny scripts and batchfiles from the Windows installation script and that was the moment I really stumbled into the annoyance of Windows backslashes. The backslash is Unix' escape character, which makes it particularly difficult to handle. To write a Windows batchfile I decided to use the following sequence:
echo -n "bg_start ..\..\bin\sh.exe --login -c wwf2pdf" > /usr/share/wwftk/bg_wwf2pdf.bat
echo -e "\r" >> /usr/share/wwftk/bg_wwf2pdf.bat

The -n switch suppresses the terminating linefeed, while the -e switch enables the expansion of special control sequences. Which in this case results in the familiar carriage return-linefeed sequence.

The BGstart utility is required to suppress the launching of the sh window, which is featured so prominently in some ported FOSS programs. A few shortcuts and we're done. The icons for the shortcuts were converted online. Quickly and painlessly. And then it ran. Or did it?

The backslashes were back with a vengeance. Zenity returns a Windows path, which is gracefully handled by MSYS, but when it was displayed all backslashes disappeared or made the messagebox look very queer.

I never wanted special Windows versions of the scripts, the idea was that one version runs anywhere. So, how could I transparently handle this one? The answer was one single line:
SHOW=`echo "$3" | sed 's|\\\\|/|g'`

This one converts any embedded backslashes to slashes. Why slashes, you may ask. Well, first it settles the matter once and for all and second, it reminds the Windows user he owes this functionality to a Unix script.

Of course, there are many things that could be improved. A true Windows installer may make it all much easier for the casual user, but again, I'm not prepared to put so much effort into a skill I rarely use. Furthermore, if people think it is important they will step up and solve it. Because this is FOSS.

I'm now looking for someone who does the port to OS/X, so we cover the entire spectrum. If only to prove that the community does a better and faster job than a closed source company..


Anonymous said...

Nicely done! Great solution for "porting to Windows".

Quite interesting, last month I had to provide a script for Windows too*, and I combined Visual Basic Script, VBA (Excel), AutoIt (comes with Scite!), MSDOS-batch (comes with "findstr", a Windows-clone of grep it seems) and awk.exe and sed.exe from UnxUtils.

"Why?" you may ask. Simple put, because it consists of completely portable components, so it doesn't require anything to be installed at all. Moreover, AutoScripts can be compiled to .exe's, nowadays this is done by default.

Of course, I had PortablePython as a backup, but I didn't use it.

So it's surprising to see there's a whole other way to do this!

*The script was: Concatenate .xls files of certain directories, insert filename and do some filtering.

Tony Miller said...

In addition to MSYS and AutoIT (highly recommended, BTW, it has GUI controls like Zenity and the end result can be a self contained EXE), I have found that the GnuWin32 series of utilities are quite excellent. Their installers do things the "Windows Way", and all of the bins are in one directory, so you only have to add one path to your system environment to get all the utilities to work without specifying a path. Their collection is quite large, and includes command line tools as well as programming libraries. Hope this helps!