Convert Linux into an Embedded System

Hi all
Running Linux on your embedded system can be great for giving you a stable base, lots of tools, and you can often solve problems with a quick google search. I define embedded as any custom-built computer system, not only tiny devices. In many cases you can develop on the computer itself for ease of development, however in other cases you will need to be able to set up a cross compile environment (for example if you have a slow ARM computer and you want to compile things on your fast Intel i7 processor).

In this post I am not going into details on any topic, this is just a high level list of things I do to my Linux systems. I do provide links for more information is some cases.

Linux is great since you can modify just about anything, it is free, you can strip out components you do not need, you can configure just about anything (including the scheduler). However some of the downsides are that you can configure just about anything and break something. Linux also tends to be a little behind when new technologies are released. Common problems have been wireless cards, SSD’s, firewire and USB3. Linux is in essence just a kernel and then there are operating system distributions build upon it. Some of the common distros are Debian, Ubuntu (I know this is Debian, but this is the flavor that is very common, supported, and I often use), Fedora, Suse, and Wind River.

A common question is about if you need a real-time version of Linux. Most people really do not need real-time Linux, you need to look at your system and determine how critical it is to have guaranteed timing that an event will happen within a specific time allotment, usually the real answer is that you do not need a real-time system. I heard a speaker from Space X who said they mostly just use Ubuntu and not any real-time system on their rockets. If you do need real-time Linux there are several variants including Wind River (costs $$$) and RTEMS (FREE).

There are many people who come from the world of servers who want to setup virtual machines (VM’s) and use the Linux VM’s to run the robot. I generally don’t like this. I have had many reliability problems from VM’s that are constantly being turned on and off. I have also had VM’s randomly not recognize devices/ports that are then recognized with a reboot. Finally besides for reliability you take a large performance hit since their is a lot of overhead in running a VM. The notion that you can have a VM on the robot and then a VM on your desktop for development usually does not work since they do not stay in sync. If you are mass producing a system then you should just create a master drive or image, and clone that to all of the other drives; a VM is not needed.

Here is a rough list of steps for how to convert your Linux system into an embedded system:

1. Choosing a filesystem. Select a volatile RAM drive for maximum reliability (and hardest development cycle), For a generic drive use journaling for reliability if the system crashes or shuts down not cleanly. In Linux this generally means EXT3 or EXT4.
2. Set OS as read-only so you can handle system crashes. Click here for more details.
3. Set bootloader to not stop on failure. This is especially true for new version of Ubuntu, if the computer does not shut down cleanly it will halt at the bootloader till you press a key. Click here for more details.
4. Set BIOS (I know this is not technically Linux) to autostart when power is applied (The default is to use the switch on the computer).
5. While I am in the BIOS I will often enable some of the options to speed up to boot time. Including reducing the length of the countdown before booting.
6. If large amounts of storage needed use an SSD for the OS. And a spinning disk drive or second SSD for the data.
7. Change the default setting to not do a disk scan after X days or X boots. If your computer decides to do those scans you tend to stand around thinking your robot is broken, and then after it finally boots you wonder how it fixed itself.
8. Have a system watchdog. This will let you detect if your software crashes so you can restart the system. At minimum you can add a software watchdog. Even better is to purchase a computer with an integrated hardware watchdog.
9. Create a script in /etc/init.d/ for starting your process. Any paths you use should be absolute and not relative paths. The newer way to do this is with upstart, I just have not really started to use upstart yet.
10. Add symlinks to your /etc/init.d/ in /etc/rc*.d for the code you want to run at a given runlevel. I usually put a symlink into /etc/rc2.d/, /etc/rc3.d/, and /etc/rc5.d/. The other way to do this is to just edit the /etc/rc.local file.
11. Add /etc/udev entries for mapping devices (such as USB). This can sometimes be a tricky process that needs some experimentation with to get right. Click here for more information
12. Add other drives to /etc/fstab so the drives come up the same with proper mount points and permissions every time. One thing to watch out for is if the drive is not available fstab might panic and not boot fully. If you think your drive might not be available (or you just want to be cautious) you can set nofail as one of the options.
13. Disable all software updates (except maybe security updates). I have seen many times where a software update will break code that has been working for a long time.
14. Good tools to install are ssh (or openssh) server and rsync for managing and updating code.
15. Use GCC tool chains when possible. Do NOT mess with other tool chains unless you have a really really really good reason.
16. Use top or htop to monitor system usage after loading your code. I try to only use around 75% of CPU and memory to allow for mostly repeatable code timing.
17. If I am using a wired network (which I almost always do) I try to put my settings in /etc/networking/interfaces and not in the GUI network manager. The reason for this is that it is easier to remotely change the setting (I use SSH for everything) if you ever need to change them in the future. I rarely use wireless in an embedded application for reliability reasons. If I am using wireless I will often use the GUI out of laziness, but there are ways to still use config files for wireless (which is the better way).

There you have it, my list of things I do to stabilize and prepare a fresh Linux install to work in an embedded application.

What other steps do you take? Leave it in the comments below.

Liked it? Take a second to support David Kohanbash on Patreon!