Sunday, October 20, 2013

Hardening your Raspberry Pi

When I had traded in my Microsoft box for a Linux box, I wanted it to look and behave like the large $100K Unix boxes I had had the pleasure to work with before. For instance, I bought a tape unit to make backups. It was terribly slow, but I kind of liked the whirring of the tape when you entered rewind at the console. That was cool.

So I was overjoyed when my brother came up with an old Wyse25 serial terminal. It took some fiddling with serial cables, inittab and settings, but somehow I got the beast to cooperate with my Linux box. It silently displayed kernel messages for years.

Sometimes, when my box hang, I logged in to shut it down properly. And at one time it started to spit out "Sense errors" from my SCSI drive. That got my attention. I backed it up and restarted - but it never came up again. Fortunately, my data was safe.

But after a decade, the Wyse25 began to behave erratically. The screen blinked and became unreadable at times. Most of the time a good whack fixed it - until the moment that was no longer sufficient. Maybe it was the dust that had accumulated over time, I dunno. Anyway, I needed a replacement.

But nowadays these things are hard to get. They are out of production and even a refurbished one would set me back a few hundred quid - and I needed it shipped overseas. That wasn't going to be cheap because these things are heavy, darn!

After a while I decided a Raspberry Pi and a cheap monitor would do the job just fine, setting me back about 200 Euros. I already had several parts lying around, like a keyboard and a mouse. It would not only allow me to show kernel messages, I could use it as a cheap workstation as well. That sounded like a good deal, so I ordered one.

It came in after a few days. Setting it up was a breeze (although I could not get it to accept the full 1080p resolution - but who cares) and within 20 minutes the kernel messages were scrolling down the screen. I pimped my 16G SD image with LibreOffice and a few other packages that seemed useful at the time and was quite happy with my low-cost workstation.

I put the monitor in "ECO-mode", so the whole contraption was using a mere 18W (which was 10W less than my Wyse25). Of course the Pi was on 24/7, because I wanted it to be available straight away - booting it every time would defeat the whole purpose. Problem solved. I thought.

After a few months I wanted to demonstrate the thing to a friend of mine, so I started up LibreOffice. Nothing. I rebooted it. It came up, but with errors. Fortunately ssh was still working, so it was not too big a problem. Until another reboot several months later when it just showed the dreaded kbd> prompt. That was no good..

I started googling and found out I had become the victim of "SD card corruption". It seemed to be quite common. Fortunately, there are two partitions on a Raspi SD card and my config.txt was on the first VFAT partition, so I could refrain from setting up my monitor all over again. This time I used another, smaller class 10 card and concentrated on its primary purpose - which was a console. Within half an hour I was up and running again.

But where did this corruption come from? I had been using SD cards for years (e.g. on my EeePC 701) and had never experienced corruption. Good powersupply? Got it! Overclocking? Not me. Good quality SD card? Sure! Power failure? Not that I can remember. Pulling the powercord? Are you nuts - I was an system admin! Still, I didn't want to repeat that procedure over and over again. So what could I do?

There are several things I don't understand about the Raspbian distribution:

  • Why is there no separate /home partition;
  • Why is there no separate swap partition;
  • Why use a journaling filesystem, EXT4.
With a bit of tinkering, one could even mount the root filesystem read-only and temporarily remount it read-write if one installed new packages. But - that was not the case. I had to work something out myself.

Note that these instructions are for the 2013-09-25 Raspbian distribution, which has already been tweaked to minimize SD card corruption by setting noatime to the root filesystem and moving a whole lot of stuff to tmpfs. But more could be done.

The first thing was to remove the swapfile. A box that just does a little ssh would probably not need it. That was simple.

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

The next thing was to move /tmp to memory. Just issue:

sudo vi /etc/default/tmpfs

And then set this parameter:


Finally, I wanted the logfiles in memory too. Note that they quietly disappear every time you shutdown, so they are of very little use when your Raspi has crashed. Just issue:

sudo vi /etc/fstab

And make sure it looks like this:

proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults,noatime 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
logfs /var/log tmpfs size=10M,noatime 0 0

However, this is not enough to get rid of all writes to the SD card. resolv.conf will still be rewritten each and every time - and of course, we still got the journal. But - it's not too bad for a start.

The Raspi has been working without a hitch for several weeks now. Rebooting was not a problem. But I still feel the whole story will pop up again some time or another, so I have been contemplating what my next move will be. The most painless one is to do what I suggested before: make the root filesystem read-only. There is even a special Raspbian for that one.

You could also try using a USB stick or USB harddisk as well. Just be sure that the Raspi gets enough power, which often means you need a powered USB hub. Another drawback is that you have to fork out a load of cash for this solution and that I've read reports about corrupted USB sticks as well. No guarantees here, so I can't really recommend that.

The bottom line is that the Raspi can boot from a whole bunch of devices, so you're free to make your own variant of Raspbian (e.g. on EXT2) or even try booting from NFS. It's all not too difficult as I will show you in the next post, where I'll explain how to move Raspbian to another medium - and how to emulate it. Which I do as well, because I don't feel happy developing on a physical Raspi for obvious reasons.

Although some may claim I have given you the impression that the Raspi is unreliable or a money pit, be assured that is clearly not my intention. I think it's a very nice machine with an incredible value-for-money, but it may need some tinkering to make it work for you in a particular situation, especially if you use it for production purposes - like a server that has to run 24/7.

And that's the beauty of the whole concept: you can!

Friday, May 4, 2012

My life with Coherent, part 2

In the first post of these series I've shown you it's possible to make Coherent 4.2.10 run under the newest version of QEMU. Now it's time to take it one step further.

I've had permission from Robert Swartz himself to distribute the first perfectly legal Coherent 4.2.10 QEMU image as long as you comply with the following conditions:
You may use the software on this image free of charge for personal, non-commercial use. You may NOT redistribute this image or the software it contains without written permission from the copyright holder. The software is provided on an "as is" basis without warranty of any kind.

Getting started
Download and unpack the archive. Included are two images, coherent.public.img and fat16.dd. The first one contains the Coherent image, the second one an empty floppy. If you want to, you can start them right away:
qemu-system-i386 -hda coherent.public.img -fda fat16.dd -m 16
First, don't try to crank up the amount of memory, Coherent won't use it - at least not this kernel. Second, you just have to be patient to attempt this, because it may take up to 10 minutes before Coherent is up and running. The good news is that you can monitor how far it is by pressing [CTRL]-[ALT]-2 in QEMU and typing:
info blockstats
If rd_bytes reaches 600,000 you can probably login. There are two login entries, the obvious root with password rootroot and a normal user named habe with password habehabe. Pick any one.

When you've logged in you might want to try the usual stuff like ls -l or ps -eaf, maybe you're even so bold to start up vi or cc. In any case, you will find that everything is painfully slow. If you want to get out, you can do so, but you'll have to be root in order to do it properly. Just issue:
cd; /etc/shutdown halt 0
Note it doesn't shutdown immediately! You'll have to wait until Coherent tells you it's safe to power down. Be patient! You can also exit QEMU without shutting down, but in that case you may have corrupted your image. And believe me, at these speeds you do not want to run fsck..

So that's it? Coherent runs, but it is completely unusable? No, not quite. Coherent does some disk caching, so after a while it becomes responsive. That is: until you reboot. But how can you avoid a reboot? Well, it's quite simple. QEMU allows you to make snapshots, so if you need Coherent you don't boot it, but simply load the snapshot.

QEMU only allows snapshots when you're using the so called "qcow2" image type, so we have to convert the files you just downloaded. That's not too hard, since QEMU comes with a handy utility for that. Just issue:
qemu-img convert -c -O qcow2 coherent.public.img Coherent.dsk
qemu-img convert -c -O qcow2 fat16.dd fat16.dsk
The latter may give you a warning, but don't worry, we'll only use it as a placeholder. Ok, we're done, let's start it up again:
qemu-system-i386 -hda Coherent.dsk -fda fat16.dsk -m 16
Agreed, you're in for the same wait. Sorry for that. But I promise you next time it'll be much faster. Just stay with me for a little while longer. When Coherent comes up, log in as usual. Then press [CTRL]-[ALT]-2 to enter the monitor. Now type:
savevm test0
This will save the current state of Coherent in a QEMU snapshot, so you'll end up here next time you start it. Return to the emulation by pressing [CTRL]-[ALT]-1 fiddle around a little bit, start up your favorite commands a few times and then save a new snapshot:
savevm test1
Shutdown Coherent properly and start it again, but this time with the command:
qemu-system-i386 -hda Coherent.dsk -fda fat16.dsk -m 16 -loadvm test1
Ok, that's more like it, isn't it? Ok, shut it down again. This time you'll be able to start it up much faster. Let's try something fancy now and invoke QEMU with:
qemu-system-i386 -hda Coherent.dsk -fda fat16.dsk -m 16 -serial telnet:localhost:4444,server,nowait -loadvm test1
Log in as root and type:
enable /dev/com1l
Now start up a terminal session on your host and type:
telnet localhost 4444
Yes, you can login to Coherent from your host Operating System. But that's about everything you can do as far as networking under Coherent is concerned.

There is a full TCP/IP stack available for Coherent, but you'll need a special kernel in order to make it work - which doesn't seem to be available. So I haven't succeeded to do much more in this area. If you have, please leave a comment.

Installing Coherent
Maybe you'll think this small system is so much fun that you want to give it a permanent place on your desktop. That is where the script comes in. It works with both KDE (KDialog) and Gnome (Zenity) and it makes it easier to launch the QEMU/Coherent combination. The only thing you have to do is to state where your disks are located:
If you want to exchange files between Coherent and your host, you'll have to change /etc/fstab as well:
/opt/qemu/fat16.dd /mnt/coherent msdos rw,user,noauto,loop 0 0
Note this is the uncompressed, original file! You can mount the uncompressed Coherent image as well (as sysv), but you won't be able to write to it. I don't think I'll have to tell you how to mount a floppy image, do I? Once the VM is up and running you can exchange the virtual, empty floppy for the raw version by switching to the QEMU monitor and issuing:
change floppy0 fat16.dd
However, while the raw floppy is online you can't make a snapshot. When you want to take a snapshot you simply return to the dummy floppy by issuing:
change floppy0 fat16.dsk
Ok, now we've cleared that one we can concentrate on the real issue here: how do we get any files from the floppy? That's not as difficult as you think. You'll only need to edit one file, /etc/default/msdos. Make sure the appropriate section reads:
# This is for a system with a 3.5 inch A drive and a 5.25 inch B drive
And you're ready to rock 'n roll. There is a whole slew of MS-DOS related commands at your disposal like:
  • dosls
  • doscp
  • dosrm
  • dosrmdir
E.g. copying the /etc/default/msdos file to the floppy is simply:
doscp /etc/default/msdos a:
And copying it back is done by:
doscp a:/msdos /etc/default
Well, that ain't rocket science, is it? We can use it right away to fix some Y2K issues. Yes, this version of Coherent won't go past the 31st of December 1999, but it is easily fixed by ATclock and date. Copy them to the /bin directory and reboot. That'll fix it.

You may also be tempted to try another kernel. Several ones are available here. Just copy it to the root and hard-link it to autoboot, e.g.:
ln cohat0 autoboot
Note you can also select a kernel when starting up - just press [SPACE] immediately. Coherent will answer:
If installing COHERENT, please type "begin".
But it is actually a prompt for the kernel name. Obviously, if you type autoboot it will boot the default kernel - just in case you don't know how to get out of there ;-)

Final notes
  • Although Coherent comes with a full fledged C compiler, it won't compile your ANSI-C sources, since it is strictly K&R C. However, GCC - and all the GNU stuff that comes along with it - is available. Still, I like the little beast, since it makes small and fast compilants.
  • Coherent does have virtual consoles. Press [CTRL]-[NUM0], [CTRL]-[NUM1], etc.
  • The archives of comp.os.coherent are compelling reading, featuring Linus Torvalds on who's got the best OS.
  • Additional notes and links on the subject are appreciated. Just leave them in the comments.

Useful links
Good sources for additional software are:
Additional information can be obtained:

There have been consistent reports on Coherent being considerably faster on QEMU 1.7 and up. Well, I tested it and even if there may be some truth to it, it's not very significant nor consistent. If you can deal with it, convert your QCOW2 image to a raw one, because that will give you the required speedup.

But the biggest drawback is that your C compiler will go haywire, spitting out errors on perfectly fine include files. This behavior was still present on the QEMU 2.0.2 version, although slightly less prominent.

The last QEMU version known to work flawlessly is 1.4.

Saturday, February 25, 2012

Hollywood still doesn't get it

The Dutch Anti-Piracy organization "Brein" has decided to sue ISP's UPC, KPN, T-Mobile and Tele2 after their recent victory in court. I really don't understand why. First of all, blocking a handful of IP addresses hasn't had any effect at all.

But most of all, if you really want to fight piracy you got to have support from the public. And that support is crumbling with each and every effort to enforce compliance to that 10 minutes of pesky messages you get when you insert a "legal" DVD. Yeah, you name them all: SOPA, PIPA, ACTA or whatever they may be called. Crowds are cheering on the streets and can't wait to have them ratified.

Metallica, once one of the most fierce fighters of piracy, has seen the sign on the wall and radically changed its position. These guys are not stupid. They know their stand on piracy affected their popularity, so they took the only decision they could take.

I've never been a downloader. I simply don't like the hassle that comes with it. But I have friends who are. One of them is a gray haired hippie, who also happens to be a record collector. Consequently, a burned CD has little value to him. But he is also a great fan of vintage science fiction movies. You hardly find those movies in the local stores and when you do the prices are outrageous. So every now and then he ordered one at Amazon. When you add all the additional costs they're not quite cheap either.

So when he wanted "to go on the Internet" in the early 2000's I told him to buy a Mac and get XS4ALL. A whole new world opened up to him. One night when we were having an beer and he told me how hard it was to get a decent copy of "Jason and the Argonauts" for a reasonable price. So I introduced him to the world of torrents, clearly stating that although downloading wasn't illegal in the Netherlands, it wasn't quite legal as well.

"Unlawful" isn't black and white down here, but has quite a few shades of gray. E.g. contrary to popular belief marijuana isn't legal here, it's just not.. completely illegal. It's - as we Dutch say - "condoned", which means you aren't prosecuted.

We quickly found a viable torrent and started downloading. It trickled down at about 10 kB/s, but he wasn't in a hurry. And a day or so later it was there. The weeks that followed he went into a kind of download frenzy, but then it settled down. I mean, they don't make movies like that anymore and the more recent ones you can get in the shop.

We met on the street a few weeks ago and we quickly landed on the subject of the recent Ziggo/XS4ALL verdict. He was furious. "Who are they to tell me where I can or cannot surf?!" he said. I told him he still could. I told him to take a look at my blog and simply follow the links.

That evening he phoned me to say everything worked fine and he was currently downloading "Captain America, the first avenger". "I'm gonna boycott them!" he said "I spend about 20 Euros a week on DVD's and if this is how they're treating me, they're gonna lose a customer! That thing is in the store for about 15 Euros - that's too expensive for my taste, but I'm gonna watch it tonight! You won't believe the download speed I'm getting, about 500 kB/s!! It's even got Dutch subtitles!"

I wasn't surprised. Everyone knows that the more popular a title is, the more seeders and leechers are offering it, which really helps to speed up the download. He had already downloaded "The Thing 2", "The Green Lantern" and I'm sure more were coming. So that is in effect what Mr. Kuik, spokesman of "BREIN", achieved. And he's making himself more popular by the minute.

The point is that the entertainment industry seems to be unable to listen to their best customers. They want the world to play by their rules, but every enterpreneur knows that's a very bad business model. Studies prove that the entertainment industry can survive and even make money, but they simply have to start to use their brains ("BREIN" means "BRAIN" in Dutch).

One of these pioneers is "Iron Sky", that partly uses "crowd funding" to raise money. And they will offer the movie for download once it has been in the theatres. Now that's creative thinking. I won't say it will work, but at least they're trying. Most importantly, they have the support of the community.

In contrast, the music industry have tried to tie down their customers with DRM. Needless to say they failed miserably - as I predicted - and nowadays it is very hard to find a CD or download with DRM. It simply doesn't work that way, despite state-of-the-art technology and elaborate schemes to "hide" the disadvantages from DRM to the public.

Now they're relying again on technology to fight piracy, but this time the technology is not in their hands, so it is even easier to find a way around it. Technology has always been a double edged sword for Hollywood. The introduction of the TV almost brought it to its knees, CGI on the other hand, produced some of its most famous blockbusters.

However, it has to realize that the Internet is nothing more than the 21st century equivalent of the TV. It can't be controlled and you can't legislate it away. Hollywood will have to change its game. Mikhail Gorbachev once said "Those who come too late will be punished by life itself".

Hollywood may not realize it, but the Internet is not the last challenge it has to face. In 15, 20 years, may be sooner, every kid with a computer will be able to create his own Hollywood grade movies. There will be digitilized Marilyn Monroes, James Deans, Humphrey Bogarts, landscapes from all over the world, cars from every era. Of course, most of these movies will be very, very bad. But some of them will be great. Most important of all: they will be free. And when they're not ready for the Internet, they aren't prepared for that.

With the turn of the millennium I expected "20th Century Fox" to change its name. It didn't. Now I understand why.