Sunday, October 20, 2013

Hardening your Raspberry Pi

When I had traded in my Microsoft box for a Linux box, I wanted it to look and behave like the large $100K Unix boxes I had had the pleasure to work with before. For instance, I bought a tape unit to make backups. It was terribly slow, but I kind of liked the whirring of the tape when you entered rewind at the console. That was cool.

So I was overjoyed when my brother came up with an old Wyse25 serial terminal. It took some fiddling with serial cables, inittab and settings, but somehow I got the beast to cooperate with my Linux box. It silently displayed kernel messages for years.

Sometimes, when my box hang, I logged in to shut it down properly. And at one time it started to spit out "Sense errors" from my SCSI drive. That got my attention. I backed it up and restarted - but it never came up again. Fortunately, my data was safe.

But after a decade, the Wyse25 began to behave erratically. The screen blinked and became unreadable at times. Most of the time a good whack fixed it - until the moment that was no longer sufficient. Maybe it was the dust that had accumulated over time, I dunno. Anyway, I needed a replacement.

But nowadays these things are hard to get. They are out of production and even a refurbished one would set me back a few hundred quid - and I needed it shipped overseas. That wasn't going to be cheap because these things are heavy, darn!

After a while I decided a Raspberry Pi and a cheap monitor would do the job just fine, setting me back about 200 Euros. I already had several parts lying around, like a keyboard and a mouse. It would not only allow me to show kernel messages, I could use it as a cheap workstation as well. That sounded like a good deal, so I ordered one.

It came in after a few days. Setting it up was a breeze (although I could not get it to accept the full 1080p resolution - but who cares) and within 20 minutes the kernel messages were scrolling down the screen. I pimped my 16G SD image with LibreOffice and a few other packages that seemed useful at the time and was quite happy with my low-cost workstation.

I put the monitor in "ECO-mode", so the whole contraption was using a mere 18W (which was 10W less than my Wyse25). Of course the Pi was on 24/7, because I wanted it to be available straight away - booting it every time would defeat the whole purpose. Problem solved. I thought.

After a few months I wanted to demonstrate the thing to a friend of mine, so I started up LibreOffice. Nothing. I rebooted it. It came up, but with errors. Fortunately ssh was still working, so it was not too big a problem. Until another reboot several months later when it just showed the dreaded kbd> prompt. That was no good..

I started googling and found out I had become the victim of "SD card corruption". It seemed to be quite common. Fortunately, there are two partitions on a Raspi SD card and my config.txt was on the first VFAT partition, so I could refrain from setting up my monitor all over again. This time I used another, smaller class 10 card and concentrated on its primary purpose - which was a console. Within half an hour I was up and running again.

But where did this corruption come from? I had been using SD cards for years (e.g. on my EeePC 701) and had never experienced corruption. Good powersupply? Got it! Overclocking? Not me. Good quality SD card? Sure! Power failure? Not that I can remember. Pulling the powercord? Are you nuts - I was an system admin! Still, I didn't want to repeat that procedure over and over again. So what could I do?

There are several things I don't understand about the Raspbian distribution:

  • Why is there no separate /home partition;
  • Why is there no separate swap partition;
  • Why use a journaling filesystem, EXT4.
With a bit of tinkering, one could even mount the root filesystem read-only and temporarily remount it read-write if one installed new packages. But - that was not the case. I had to work something out myself.

Note that these instructions are for the 2013-09-25 Raspbian distribution, which has already been tweaked to minimize SD card corruption by setting noatime to the root filesystem and moving a whole lot of stuff to tmpfs. But more could be done.

The first thing was to remove the swapfile. A box that just does a little ssh would probably not need it. That was simple.

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

The next thing was to move /tmp to memory. Just issue:

sudo vi /etc/default/tmpfs

And then set this parameter:


Finally, I wanted the logfiles in memory too. Note that they quietly disappear every time you shutdown, so they are of very little use when your Raspi has crashed. Just issue:

sudo vi /etc/fstab

And make sure it looks like this:

proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults,noatime 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
logfs /var/log tmpfs size=10M,noatime 0 0

However, this is not enough to get rid of all writes to the SD card. resolv.conf will still be rewritten each and every time - and of course, we still got the journal. But - it's not too bad for a start.

The Raspi has been working without a hitch for several weeks now. Rebooting was not a problem. But I still feel the whole story will pop up again some time or another, so I have been contemplating what my next move will be. The most painless one is to do what I suggested before: make the root filesystem read-only. There is even a special Raspbian for that one.

You could also try using a USB stick or USB harddisk as well. Just be sure that the Raspi gets enough power, which often means you need a powered USB hub. Another drawback is that you have to fork out a load of cash for this solution and that I've read reports about corrupted USB sticks as well. No guarantees here, so I can't really recommend that.

The bottom line is that the Raspi can boot from a whole bunch of devices, so you're free to make your own variant of Raspbian (e.g. on EXT2) or even try booting from NFS. It's all not too difficult as I will show you in the next post, where I'll explain how to move Raspbian to another medium - and how to emulate it. Which I do as well, because I don't feel happy developing on a physical Raspi for obvious reasons.

Although some may claim I have given you the impression that the Raspi is unreliable or a money pit, be assured that is clearly not my intention. I think it's a very nice machine with an incredible value-for-money, but it may need some tinkering to make it work for you in a particular situation, especially if you use it for production purposes - like a server that has to run 24/7.

And that's the beauty of the whole concept: you can!


Tiger said...

I was wondering when the mentioned new post will be published ?

It's all not too difficult as I will show you in the next post, where I'll explain how to move Raspbian to another medium - and how to emulate it.

Tiger said...

I was wondering when the mentioned follow up article will be published ?

It's all not too difficult as I will show you in the next post, where I'll explain how to move Raspbian to another medium - and how to emulate it.

The Beez' said...

First of all, I wrote this thing right after I took the measures. I had to see if it were enough. Second, since this is a private site - and not a mag - it depends on "when time is available". You may not think too much of a page like this, but a well-writen post takes me a day!

Jon Russell said...

I have the same issue ... did your changes work ? have you had any issues since ? I notice this was posted in Oct 2013 ? almost 6 months ago ... I'm getting about 3-4 months out of a vanilla setup, before I lose the SD filing system... 6 months stability, would be a huge improvement ... Thanks. Jon.

The Beez' said...

Well, first of all, I switch from a 16GB card to a 4 GB card - and didn't expand it. Second, I applied all changes listed here plus a few minor ones (like disabling the Samba setup). It doesn't still eradicate all writing, but substantially lowers it. Mine is used as a console for a larger machine (as presented here) so it doesn't get much of a workout. BUT: it's OK. No more garbling of the SD card (knock on wood).

Anonymous said...

I have 4 rpi boxes around here and sdcard corruption has been a constant nuisance. We are prone to electricity glitches, but even having them run off a UPS hasn't made the problem go away.

I found that the only workable solution is to have the root file system on NFS. This works a treat - no problems for a year. I haven't noticed any performance problems.

Anonymous said...

If you are looking for something like this in 2017, Alpine Linux is a good starting point: It is tiny, stable, server-focused, and designed to run from RAM.