The goals of this project was to build a low-power, small form factor machine that runs OpenBSD and acts as a firewall/router in a home network or small business setting. This page walks through the hardware I chose and the process I use to get OpenBSD running on the CF card.
Table of Contents
The design has gone through two generations of hardware now.
Small Form Factor Shuttle System
The first firewall I built was in 2004 using a Shuttle SV25 small form factor case and a VIA C3 Ezra 933MHz CPU. The machine was passively cooled, had 256MB RAM a Phobos P430/TX quad port fast ethernet NIC and a 256MB compact flash card.
In the fall of 2007 the system started randomly crashing and causing kernel panics. After some hardware troubleshooting, I decided it would be better to simply replace the entire thing.
Rack Mount Appliance
- Intel 852GM chipset
- Up to 2GHz Pentium-M processor
- Up to 1.25GB DDR226 memory
- On-board compact flash slot
- 2x Broadcom BCM4401B1 fast ethernet NICs
- 2x Broadcom BCM5788 gigabit ethernet NICs
- Hardware watchdog
- Space for 3.5″ or 2.5″ IDE drive
- 1RU form factor
This hardware has performed well but it does have some limitations:
- No on-board SATA which means connecting a modern hard drive or even a solid state drive is out of the question.
- There are no ports on the outside of the box for connecting a keyboard and monitor. The only connectors are pins located on the board. You have to take the top cover off the unit and connect these little jumper cables to the board which let you plug your PS/2 keyboard and VGA monitor in.
- The front-facing serial port does not work well enough to use as a console port. You can either: 1. Enable the console port function in the system's BIOS in which case you can interact with the system during POST, get into the BIOS, etc, but lose all console function as soon as the OpenBSD bootloader loads. Or, 2. Disable the console port function in the BIOS and enable console on "com0" in OpenBSD in which case you cannot interact with the BIOS nor can you enter any commands at the "boot>" prompt but you can see the boot messages and login on the console when the machine comes up.
- Neither of these options is perfect, especially when you're doing maintenance on the box and need to either boot a rescue kernel from the "boot>" prompt or get into the BIOS to change the boot order.
- The Broadcom BCM4401 ethernet chipset is not highly regarded. It's well known the card misbehaves in systems that have > 1GB of memory in them. See OpenBSD bugid 6074.
The base operating system is OpenBSD, but it requires a few changes to adapt it to the router. These are the major changes:
- A number of RAM drives need to be created and populated with files at system boot
- Some important symbolic links need to be created so that directories located on a RAM drive do not lose their contents on system reboot
Working with RAM Drives
RAM drives are created for areas of the filesystem that are written to frequently such as /dev and /var. Since Compact flash drives can only be written to a finite number of times, using RAM drives will extend the life of the flash drive. The following RAM drives (aka, Memory File Systems) are created:
Filesystem 1K-blocks Used Avail Capacity Mounted on mfs:21550 1711 24 1602 1% /dev mfs:18708 19759 1 18771 0% /tmp mfs:26333 19759 4392 14380 23% /var
The /dev file system is populated on boot using MAKEDEV(8). MAKEDEV normally resides in /dev, but because /dev is an empty RAM drive, MAKEDEV has been moved to /sbin.
The /var file system is a tricky one to store on a RAM drive because there are files stored there that need to be available after the system reboots. As things are now, once the system starts up, /var will be an empty RAM drive. The solution is to populate it from a "skeleton" directory (/var.skel) which contains the basic directory structure and files needed to get /var into a usable state. When the system is installed, the /var directory is renamed to /var.skel (thereby creating our skeleton structure) and an empty /var directory is created to be the mount point for the RAM drive.
Creation of the RAM drives and population of /var from /var.skel are handled by making some changes to /etc/rc. Immediately after the root file system is mounted:
echo -n "Making memory filesystems... " echo -n "/dev " mount_mfs -i 512 -s 4096 swap /dev cd /dev && /sbin/MAKEDEV all chmod 755 /dev echo -n "/tmp " mount_mfs -s 40960 swap /tmp echo -n "/var " mount_mfs -s 40960 -P /var.skel swap /var echo
The values given to mount_mfs are chosen carefully.
- /dev holds a lot of very small files which consume a lot of inodes (relative to the data capacity needed) so it's necessary to increase the number of inodes to one per 512 bytes of data space (default is one per 8192 bytes). If you run out of inodes when MAKEDEV is running, it's very likely the system will not boot.
- The size of the /dev file system has been set way higher than what is needed to store the actual device files, however it is necessary because the number of inodes available is proportional to the size of the file system.
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mount mfs:21550 1711 24 1602 1% 1522 524 74% /dev
- You can see above that only 1% of the data capacity has been used and 74% of the inodes.
- The size of /var is set to give enough room for the skeleton files and syslogs. A 20MB size gives lots of headroom. The -P argument tells mount_mfs to populate the new RAM drive with the contents of the /var.skel directory.
- The size of /tmp is pretty arbitrary; since there aren't really any applications running on this system, very few files get written there. It's also set to 20MB.
When the system restarts, any changes that were made to any files or directories on a RAM drive are lost. This impacts two directories of particular importance: /var/cron and /var/db/pkg. The solution is to store these directories on the actual flash drive itself. A symbolic link for each directory is created back into /etc in order to accomplish this.
root@mu:~# ls -l /var/cron /var/db/pkg lrwxr-xr-x 1 root wheel 9 May 4 20:14 /var/cron -> /etc/cron lrwxr-xr-x 1 root wheel 11 May 4 20:14 /var/db/pkg -> /etc/db/pkg
Some notes on the firewall's operation:
- There is no swap space configured. Having swap on the compact flash card would not only greatly impact performance, but would decrease the lifetime of the card. There is enough physical RAM in the device that swapping shouldn't be necessary anyways (knock wood).
- The syslog daemon is setup to log in the usual /var/log location (which is on a RAM drive) as well as to a syslog server. Sending the logs to another server ensures the log archive is available even if the firewall reboots and the contents of the RAM drive are lost.
- In order to work around the bug in the Broadcom BCM4401 chip mentioned above, the amount of RAM in the system is artificially limited via the /etc/boot.conf file with this entry:
machine memory =1G
- As a result of the issues in the BCM4401 chip, the OpenBSD project has disabled the bce(4) driver in the GENERIC kernel. The driver needs to be added back and the kernel recompiled in order to use those two NICs.