Tag Archives: openbsd-pf


It’s May and that means a new version of OpenBSD is out. My SNMP MIBs have been updated for 5.1 and are available for download on the OpenBSD SNMP MIBs page.


During the OpenBSD 5.1 development cycle, I committed the CARP MIB to the base OpenBSD snmpd. The kernel sensor MIB has been in the base snmpd for a few releases now. That leaves the pf MIB which was committed to 5.1-current some weeks ago and will be present in the 5.2 release.

So, you’ve got a few options.

  1. Want to still run Net-SNMP with the extra MIBS? Go to the SNMP MIBs page and follow the directions. No change from previous versions. However, make plans to migrate away from Net-SNMP for OpenBSD 5.2.
  2. Only use the CARP or kernel sensors MIB? Use the base snmpd(8). There’s no configuration necessary, just run the daemon. The MIB files are in /usr/share/snmp/mibs/ (The pf MIB file is present there, but the implementation is not part of snmpd(8) in 5.1). You should also read my guide on what’s changed between the Net-SNMP and snmpd(8) implementations of the MIBs.
  3. Want to use the base snmpd(8) but still have a requirement for Net-SNMP? See my blog post on using both together.

Switching from Net-SNMP to snmpd for CARP, PF and Sensor Monitoring

Update: For help running both snmpds at the same time, seeĀ Net-SNMP and snmpd Coexistence on OpenBSD.

Now that OPENBSD-CARP-MIB and OPENBSD-PF-MIB have been added to the base snmpd in OpenBSD (CARP-MIB will be in 5.1-release, PF-MIB in 5.2, and the SENSOR MIB has been there since 4.5), I wanted to document the differences between these MIBs and the corresponding implementation of the MIBs that I wrote for Net-SNMP.

Both implementations provide the same set of OIDs and allow the same data to be retrieved. Whatever you were querying via Net-SNMP is available via snmpd.

What has changed is the base OID where the CARP and PF MIBs are rooted at as well as the name of certain OIDs. Continue reading Switching from Net-SNMP to snmpd for CARP, PF and Sensor Monitoring

Hitting the PF state table limit

I recently had an issue with an OpenBSD firewall where the number of state table entries was hitting the default limit of 10,000. When this limit is reached, no new state entries can be created. If you’re using “keep state”, “modulate state” or “synproxy state” on your rules or if you’re running OpenBSD 4.1 or newer (where “keep state” is the default on all rules) this could mean that:

  • You cannot make new connections through the firewall
  • You cannot make new connections to the firewall
  • You cannot make new connections from the firewall

So…. if you hit the state table limit it’s kinda bad, mmmkay?
Continue reading Hitting the PF state table limit

OpenBSD Compact Flash Firewall

The goals of this project was to build a low-power, small form factor machine that runs OpenBSD and acts as a firewall/router in a home network or small business setting. This page walks through the hardware I chose and the process I use to get OpenBSD running on the CF card.

Table of Contents


The design has gone through two generations of hardware now.

Small Form Factor Shuttle System

The first firewall I built was in 2004 using a Shuttle SV25 small form factor case and a VIA C3 Ezra 933MHz CPU. The machine was passively cooled, had 256MB RAM a Phobos P430/TX quad port fast ethernet NIC and a 256MB compact flash card.

Shuttle SV25 Inside
Shuttle SV25 Inside
Shuttle SV25 Rear
Shuttle SV25 Rear

In the fall of 2007 the system started randomly crashing and causing kernel panics. After some hardware troubleshooting, I decided it would be better to simply replace the entire thing.

Rack Mount Appliance

The current hardware being used is an Acrosser AR-M9936 network appliance (dmesg output).

  • Intel 852GM chipset
  • Up to 2GHz Pentium-M processor
  • Up to 1.25GB DDR226 memory
  • On-board compact flash slot
  • 2x Broadcom BCM4401B1 fast ethernet NICs
  • 2x Broadcom BCM5788 gigabit ethernet NICs
  • Hardware watchdog
  • Space for 3.5″ or 2.5″ IDE drive
  • 1RU form factor
Acrosser Front
Acrosser Front
Acrosser Inside
Acrosser Inside
Acrosser Bottom
Acrosser Bottom

This hardware has performed well but it does have some limitations:

  • No on-board SATA which means connecting a modern hard drive or even a solid state drive is out of the question.
  • There are no ports on the outside of the box for connecting a keyboard and monitor. The only connectors are pins located on the board. You have to take the top cover off the unit and connect these little jumper cables to the board which let you plug your PS/2 keyboard and VGA monitor in.
  • The front-facing serial port does not work well enough to use as a console port. You can either:
    1. Enable the console port function in the system’s BIOS in which case you can interact with the system during POST, get into the BIOS, etc, but lose all console function as soon as the OpenBSD bootloader loads. Or,
    2. Disable the console port function in the BIOS and enable console on “com0” in OpenBSD in which case you cannot interact with the BIOS nor can you enter any commands at the “boot>” prompt but you can see the boot messages and login on the console when the machine comes up.
  • Neither of these options is perfect, especially when you’re doing maintenance on the box and need to either boot a rescue kernel from the “boot>” prompt or get into the BIOS to change the boot order.
  • The Broadcom BCM4401 ethernet chipset is not highly regarded. It’s well known the card misbehaves in systems that have > 1GB of memory in them. See OpenBSD bugid 6074.

Operating System

The base operating system is OpenBSD, but it requires a few changes to adapt it to the router. These are the major changes:

  • A number of RAM drives need to be created and populated with files at system boot
  • Some important symbolic links need to be created so that directories located on a RAM drive do not lose their contents on system reboot

Working with RAM Drives

RAM drives are created for areas of the filesystem that are written to frequently such as /dev and /var. Since Compact flash drives can only be written to a finite number of times, using RAM drives will extend the life of the flash drive. The following RAM drives (aka, Memory File Systems) are created:

Filesystem   1K-blocks Used Avail Capacity Mounted on
mfs:21550         1711   24  1602     1%   /dev
mfs:18708        19759    1  18771    0%   /tmp
mfs:26333        19759 4392  14380   23%   /var

The /dev file system is populated on boot using MAKEDEV(8). MAKEDEV normally resides in /dev, but because /dev is an empty RAM drive, MAKEDEV has been moved to /sbin.

The /var file system is a tricky one to store on a RAM drive because there are files stored there that need to be available after the system reboots. As things are now, once the system starts up, /var will be an empty RAM drive. The solution is to populate it from a “skeleton” directory (/var.skel) which contains the basic directory structure and files needed to get /var into a usable state. When the system is installed, the /var directory is renamed to /var.skel (thereby creating our skeleton structure) and an empty /var directory is created to be the mount point for the RAM drive.

Creation of the RAM drives and population of /var from /var.skel are handled by making some changes to /etc/rc. Immediately after the root file system is mounted:

echo -n "Making memory filesystems... "
echo -n "/dev "
mount_mfs -i 512 -s 4096 swap /dev
cd /dev && /sbin/MAKEDEV all
chmod 755 /dev
echo -n "/tmp "
mount_mfs -s 40960 swap /tmp
echo -n "/var "
mount_mfs -s 40960 -P /var.skel swap /var

The values given to mount_mfs are chosen carefully.

  • /dev holds a lot of very small files which consume a lot of inodes (relative to the data capacity needed) so it’s necessary to increase the number of inodes to one per 512 bytes of data space (default is one per 8192 bytes). If you run out of inodes when MAKEDEV is running, it’s very likely the system will not boot.
  • The size of the /dev file system has been set way higher than what is needed to store the actual device files, however it is necessary because the number of inodes available is proportional to the size of the file system.
Filesystem   1K-blocks Used Avail Capacity iused ifree %iused Mount
mfs:21550         1711   24  1602     1%    1522   524   74%  /dev
  • You can see above that only 1% of the data capacity has been used and 74% of the inodes.
  • The size of /var is set to give enough room for the skeleton files and syslogs. A 20MB size gives lots of headroom. The -P argument tells mount_mfs to populate the new RAM drive with the contents of the /var.skel directory.
  • The size of /tmp is pretty arbitrary; since there aren’t really any applications running on this system, very few files get written there. It’s also set to 20MB.

Important Symlinks

When the system restarts, any changes that were made to any files or directories on a RAM drive are lost. This impacts two directories of particular importance: /var/cron and /var/db/pkg. The solution is to store these directories on the actual flash drive itself. A symbolic link for each directory is created back into /etc in order to accomplish this.

root@mu:~# ls -l /var/cron /var/db/pkg
lrwxr-xr-x 1 root wheel  9 May 4 20:14 /var/cron -> /etc/cron
lrwxr-xr-x 1 root wheel 11 May 4 20:14 /var/db/pkg -> /etc/db/pkg

System Operation

Some notes on the firewall’s operation:

  • There is no swap space configured. Having swap on the compact flash card would not only greatly impact performance, but would decrease the lifetime of the card. There is enough physical RAM in the device that swapping shouldn’t be necessary anyways (knock wood).
  • The syslog daemon is setup to log in the usual /var/log location (which is on a RAM drive) as well as to a syslog server. Sending the logs to another server ensures the log archive is available even if the firewall reboots and the contents of the RAM drive are lost.
  • In order to work around the bug in the Broadcom BCM4401 chip mentioned above, the amount of RAM in the system is artificially limited via the /etc/boot.conf file with this entry: machine memory =1G
  • As a result of the issues in the BCM4401 chip, the OpenBSD project has disabled the bce(4) driver in the GENERIC kernel. The driver needs to be added back and the kernel recompiled in order to use those two NICs.

NetPacket PERL Module Enhancements

NetPacket provides a base class for a cluster of modules related to decoding and encoding of network protocol packets. Each NetPacket descendant module knows how to encode and decode packets for the network protocol it implements. Protocols that NetPacket can encode/decode include IPv4, TCP, UDP, ICMP, Ethernet, and ARP.

I’ve written three additional modules for NetPacket that allow the encoding/decoding of IPv6, ICMPv6, and OpenBSD’s Packet Filter binary log files. I’ve also made numerous changes to existing modules, including fixing spelling mistakes, bug fixes, and documentation enhancements.

If you’re using OpenBSD, you can install the p5-NetPacket port/package which already contains the patches outlined below.

IPv6 Modules

The IPv6 modules allow for the encoding/decoding of IPv6 and ICMPv6 packets. These modules, like the modules that make up the NetPacket distribution, can be used to analyze tcpdump or Wireshark packet captures that contain IPv6/ICMPv6 packets.

Some example code might look like this:

#!/usr/bin/perl -w

use strict;
use Net::PcapUtils;
use NetPacket::Ethernet qw(:strip);
use NetPacket::IPv6;

sub process_pkt {
  my ($user, $hdr, $pkt) = @_;

  my $ip6_obj = NetPacket::IPv6->decode(eth_strip($pkt));
  print("$ip6_obj->{src_ip} -> $ip6_obj->{dest_ip} ");

Net::PcapUtils::loop(&process_pkt, FILTER => 'ip6');

This code snippet starts a packet capture and looks for IPv6 packets (the FILTER keyword). Each packet seen is passed to the process_pkt function where the source IP, destination IP, and the “next header” value are output. The output would look similar to this:

2001:618:4cfa:a00:0:0:1:a00 -> 2001:b40d:44:62:0:0:0:276 58

The perlpod documentation that comes with the modules explains all of the data fields that get decoded as well as the object methods. Use the perldoc NetPacket::IPv6 and perldoc NetPacket::ICMPv6 commands to view the documentation.

OpenBSD Packet Filter Module

OpenBSD’s Packet Filter firewall software stores its log files as a libpcap packet dump. This is a binary file format and cannot be read by humans. Additional software is needed to parse and format the log files for analysis and viewing. The PFLog NetPacket module that I’ve written allows for decoding of this binary data.

Some example code might look like this:


use strict;
use Net::Pcap;
use NetPacket;
use NetPacket::IP;
use NetPacket::PFLog;

sub analyze_dump {
  my ($user_data, $header, $packet) = @_;
  my $pflog = NetPacket::PFLog->decode($packet);
  my $ip = NetPacket::IP->decode($pflog->{data});
  print "$pflog->{action} $pflog->{dir} on $pflog->{ifname}";
  print " $ip->{src_ip} -> $ip->{dest_ip}\n";

my ($pcap_t, $pcaperr);
$pcap_t = Net::Pcap::open_offline("/var/log/pflog", \$pcaperr);
Net::Pcap::loop($pcap_t, -1, &analyze_dump, "");

This code snippet opens the PF log file /var/log/pflog, iterates through each packet in the log and prints the action taken on the packet (pass or block), the packet direction (in or out), the interface name, and the source and destination IP address. The output would look similar to this:

pass in on em0 ->

The perlpod documentation that comes with the module explains all of the data fields that get decoded as well as the PFLog object methods. Use the perldoc NetPacket::PFLog command to view the documentation.


The code is distributed as a patch to the NetPacket distribution.

Apply the patch:

% patch < netpacket-1.4.4.diff

Build NetPacket as per the INSTALL file.

If you’re an OpenBSD user, then you can simply install the net/p5-NetPacket port and be done. The patches are installed automatically (although the port may not always have the most up-to-date version of the patch).