OpenBSD Compact Flash Firewall

The goals of this project was to build a low-power, small form factor machine that runs OpenBSD and acts as a firewall/router in a home network or small business setting. This page walks through the hardware I chose and the process I use to get OpenBSD running on the CF card.

Table of Contents

Hardware

The design has gone through two generations of hardware now.

Small Form Factor Shuttle System

The first firewall I built was in 2004 using a Shuttle SV25 small form factor case and a VIA C3 Ezra 933MHz CPU. The machine was passively cooled, had 256MB RAM a Phobos P430/TX quad port fast ethernet NIC and a 256MB compact flash card.

Shuttle SV25 Inside
Shuttle SV25 Inside
Shuttle SV25 Rear
Shuttle SV25 Rear

In the fall of 2007 the system started randomly crashing and causing kernel panics. After some hardware troubleshooting, I decided it would be better to simply replace the entire thing.

Rack Mount Appliance

The current hardware being used is an Acrosser AR-M9936 network appliance (dmesg output).

  • Intel 852GM chipset
  • Up to 2GHz Pentium-M processor
  • Up to 1.25GB DDR226 memory
  • On-board compact flash slot
  • 2x Broadcom BCM4401B1 fast ethernet NICs
  • 2x Broadcom BCM5788 gigabit ethernet NICs
  • Hardware watchdog
  • Space for 3.5″ or 2.5″ IDE drive
  • 1RU form factor
Acrosser Front
Acrosser Front
Acrosser Inside
Acrosser Inside
Acrosser Bottom
Acrosser Bottom

This hardware has performed well but it does have some limitations:

  • No on-board SATA which means connecting a modern hard drive or even a solid state drive is out of the question.
  • There are no ports on the outside of the box for connecting a keyboard and monitor. The only connectors are pins located on the board. You have to take the top cover off the unit and connect these little jumper cables to the board which let you plug your PS/2 keyboard and VGA monitor in.
  • The front-facing serial port does not work well enough to use as a console port. You can either:
    1. Enable the console port function in the system’s BIOS in which case you can interact with the system during POST, get into the BIOS, etc, but lose all console function as soon as the OpenBSD bootloader loads. Or,
    2. Disable the console port function in the BIOS and enable console on “com0” in OpenBSD in which case you cannot interact with the BIOS nor can you enter any commands at the “boot>” prompt but you can see the boot messages and login on the console when the machine comes up.
  • Neither of these options is perfect, especially when you’re doing maintenance on the box and need to either boot a rescue kernel from the “boot>” prompt or get into the BIOS to change the boot order.
  • The Broadcom BCM4401 ethernet chipset is not highly regarded. It’s well known the card misbehaves in systems that have > 1GB of memory in them. See OpenBSD bugid 6074.

Operating System

The base operating system is OpenBSD, but it requires a few changes to adapt it to the router. These are the major changes:

  • A number of RAM drives need to be created and populated with files at system boot
  • Some important symbolic links need to be created so that directories located on a RAM drive do not lose their contents on system reboot

Working with RAM Drives

RAM drives are created for areas of the filesystem that are written to frequently such as /dev and /var. Since Compact flash drives can only be written to a finite number of times, using RAM drives will extend the life of the flash drive. The following RAM drives (aka, Memory File Systems) are created:

Filesystem  1K-blocks      Used     Avail Capacity  Mounted on
mfs:21550        1711        24      1602     1%    /dev
mfs:18708       19759         1     18771     0%    /tmp
mfs:26333       19759      4392     14380    23%    /var

The /dev file system is populated on boot using MAKEDEV(8). MAKEDEV normally resides in /dev, but because /dev is an empty RAM drive, MAKEDEV has been moved to /sbin.

The /var file system is a tricky one to store on a RAM drive because there are files stored there that need to be available after the system reboots. As things are now, once the system starts up, /var will be an empty RAM drive. The solution is to populate it from a “skeleton” directory (/var.skel) which contains the basic directory structure and files needed to get /var into a usable state. When the system is installed, the /var directory is renamed to /var.skel (thereby creating our skeleton structure) and an empty /var directory is created to be the mount point for the RAM drive.

Creation of the RAM drives and population of /var from /var.skel are handled by making some changes to /etc/rc. Immediately after the root file system is mounted:

echo -n "Making memory filesystems... "
echo -n "/dev "
mount_mfs -i 512 -s 4096 swap /dev
cd /dev && /sbin/MAKEDEV all
chmod 755 /dev
echo -n "/tmp "
mount_mfs -s 40960 swap /tmp
echo -n "/var "
mount_mfs -s 40960 -P /var.skel swap /var
echo

The values given to mount_mfs are chosen carefully.

  • /dev holds a lot of very small files which consume a lot of inodes (relative to the data capacity needed) so it’s necessary to increase the number of inodes to one per 512 bytes of data space (default is one per 8192 bytes). If you run out of inodes when MAKEDEV is running, it’s very likely the system will not boot.
  • The size of the /dev file system has been set way higher than what is needed to store the actual device files, however it is necessary because the number of inodes available is proportional to the size of the file system.
Filesystem  1K-blocks Used Avail Capacity iused ifree %iused  Mount
mfs:21550        1711   24  1602     1%    1522   524   74%   /dev
  • You can see above that only 1% of the data capacity has been used and 74% of the inodes.
  • The size of /var is set to give enough room for the skeleton files and syslogs. A 20MB size gives lots of headroom. The -P argument tells mount_mfs to populate the new RAM drive with the contents of the /var.skel directory.
  • The size of /tmp is pretty arbitrary; since there aren’t really any applications running on this system, very few files get written there. It’s also set to 20MB.

Important Symlinks

When the system restarts, any changes that were made to any files or directories on a RAM drive are lost. This impacts two directories of particular importance: /var/cron and /var/db/pkg. The solution is to store these directories on the actual flash drive itself. A symbolic link for each directory is created back into /etc in order to accomplish this.

root@mu:~# ls -l /var/cron /var/db/pkg
lrwxr-xr-x 1 root  wheel   9 May  4 20:14 /var/cron -> /etc/cron
lrwxr-xr-x 1 root  wheel  11 May  4 20:14 /var/db/pkg -> /etc/db/pkg

System Operation

Some notes on the firewall’s operation:

  • There is no swap space configured. Having swap on the compact flash card would not only greatly impact performance, but would decrease the lifetime of the card. There is enough physical RAM in the device that swapping shouldn’t be necessary anyways (knock wood).
  • The syslog daemon is setup to log in the usual /var/log location (which is on a RAM drive) as well as to a syslog server. Sending the logs to another server ensures the log archive is available even if the firewall reboots and the contents of the RAM drive are lost.
  • In order to work around the bug in the Broadcom BCM4401 chip mentioned above, the amount of RAM in the system is artificially limited via the /etc/boot.conf file with this entry:

machine memory =1G

  • As a result of the issues in the BCM4401 chip, the OpenBSD project has disabled the bce(4) driver in the GENERIC kernel. The driver needs to be added back and the kernel recompiled in order to use those two NICs.

Networking. Unix. Cyber Security. Code. Protocols. System Design. My Blog.