Tag Archives: openbsd

Net-SNMP 5.6.1 Missing hrSystemProcesses OID

I just upgraded a couple of machines to OpenBSD 4.9 and noticed the hrSystemProcesses OID was not being returned by Net-SNMP 5.6.1 (from the 4.9 ports/packages collection) .

jwk@theta:~% snmpwalk -v2c -c public theta .1.3.6.1.2.1.25.1.6.0
SNMPv2-SMI::mib-2.25.1.6.0 = No Such Instance currently exists 
    at this OID

I know for sure this worked on OpenBSD 4.8/Net-SNMP 5.4.2.1.

Turns out there is a bug in Net-SNMP 5.6.1 (bug 3166568) that’s causing this. It’s been fixed in their SVN tree. If you download this patch, place it into your ports/net/net-snmp/patches/ directory and recompile the port, you’ll be good to go.

Benchmarking OpenBSD: UP vs MP for “make build”

I used to have a machine sitting around that I would power up whenever I needed to build a new OpenBSD kernel or rebuild the source tree due to a patch or a new version of the OS being released. Eventually I moved that machine into a VirtualBox virtual machine running on my desktop. Recently I moved that VM into VMWare’s free ESXi hypervisor running on some pretty decent hardware. It got me wondering how much I could lower compile times by adjusting how many vCPUs were allocated to the VM. Continue reading Benchmarking OpenBSD: UP vs MP for “make build”

Hitting the PF state table limit

I recently had an issue with an OpenBSD firewall where the number of state table entries was hitting the default limit of 10,000. When this limit is reached, no new state entries can be created. If you’re using “keep state”, “modulate state” or “synproxy state” on your rules or if you’re running OpenBSD 4.1 or newer (where “keep state” is the default on all rules) this could mean that:

  • You cannot make new connections through the firewall
  • You cannot make new connections to the firewall
  • You cannot make new connections from the firewall

So…. if you hit the state table limit it’s kinda bad, mmmkay?
Continue reading Hitting the PF state table limit

OpenBSD Compact Flash Firewall

The goals of this project was to build a low-power, small form factor machine that runs OpenBSD and acts as a firewall/router in a home network or small business setting. This page walks through the hardware I chose and the process I use to get OpenBSD running on the CF card.

Table of Contents

Hardware

The design has gone through two generations of hardware now.

Small Form Factor Shuttle System

The first firewall I built was in 2004 using a Shuttle SV25 small form factor case and a VIA C3 Ezra 933MHz CPU. The machine was passively cooled, had 256MB RAM a Phobos P430/TX quad port fast ethernet NIC and a 256MB compact flash card.

Shuttle SV25 Inside
Shuttle SV25 Inside
Shuttle SV25 Rear
Shuttle SV25 Rear

In the fall of 2007 the system started randomly crashing and causing kernel panics. After some hardware troubleshooting, I decided it would be better to simply replace the entire thing.

Rack Mount Appliance

The current hardware being used is an Acrosser AR-M9936 network appliance (dmesg output).

  • Intel 852GM chipset
  • Up to 2GHz Pentium-M processor
  • Up to 1.25GB DDR226 memory
  • On-board compact flash slot
  • 2x Broadcom BCM4401B1 fast ethernet NICs
  • 2x Broadcom BCM5788 gigabit ethernet NICs
  • Hardware watchdog
  • Space for 3.5″ or 2.5″ IDE drive
  • 1RU form factor
Acrosser Front
Acrosser Front
Acrosser Inside
Acrosser Inside
Acrosser Bottom
Acrosser Bottom

This hardware has performed well but it does have some limitations:

  • No on-board SATA which means connecting a modern hard drive or even a solid state drive is out of the question.
  • There are no ports on the outside of the box for connecting a keyboard and monitor. The only connectors are pins located on the board. You have to take the top cover off the unit and connect these little jumper cables to the board which let you plug your PS/2 keyboard and VGA monitor in.
  • The front-facing serial port does not work well enough to use as a console port. You can either:
    1. Enable the console port function in the system’s BIOS in which case you can interact with the system during POST, get into the BIOS, etc, but lose all console function as soon as the OpenBSD bootloader loads. Or,
    2. Disable the console port function in the BIOS and enable console on “com0” in OpenBSD in which case you cannot interact with the BIOS nor can you enter any commands at the “boot>” prompt but you can see the boot messages and login on the console when the machine comes up.
  • Neither of these options is perfect, especially when you’re doing maintenance on the box and need to either boot a rescue kernel from the “boot>” prompt or get into the BIOS to change the boot order.
  • The Broadcom BCM4401 ethernet chipset is not highly regarded. It’s well known the card misbehaves in systems that have > 1GB of memory in them. See OpenBSD bugid 6074.

Operating System

The base operating system is OpenBSD, but it requires a few changes to adapt it to the router. These are the major changes:

  • A number of RAM drives need to be created and populated with files at system boot
  • Some important symbolic links need to be created so that directories located on a RAM drive do not lose their contents on system reboot

Working with RAM Drives

RAM drives are created for areas of the filesystem that are written to frequently such as /dev and /var. Since Compact flash drives can only be written to a finite number of times, using RAM drives will extend the life of the flash drive. The following RAM drives (aka, Memory File Systems) are created:

Filesystem   1K-blocks Used Avail Capacity Mounted on
mfs:21550         1711   24  1602     1%   /dev
mfs:18708        19759    1  18771    0%   /tmp
mfs:26333        19759 4392  14380   23%   /var

The /dev file system is populated on boot using MAKEDEV(8). MAKEDEV normally resides in /dev, but because /dev is an empty RAM drive, MAKEDEV has been moved to /sbin.

The /var file system is a tricky one to store on a RAM drive because there are files stored there that need to be available after the system reboots. As things are now, once the system starts up, /var will be an empty RAM drive. The solution is to populate it from a “skeleton” directory (/var.skel) which contains the basic directory structure and files needed to get /var into a usable state. When the system is installed, the /var directory is renamed to /var.skel (thereby creating our skeleton structure) and an empty /var directory is created to be the mount point for the RAM drive.

Creation of the RAM drives and population of /var from /var.skel are handled by making some changes to /etc/rc. Immediately after the root file system is mounted:

echo -n "Making memory filesystems... "
echo -n "/dev "
mount_mfs -i 512 -s 4096 swap /dev
cd /dev && /sbin/MAKEDEV all
chmod 755 /dev
echo -n "/tmp "
mount_mfs -s 40960 swap /tmp
echo -n "/var "
mount_mfs -s 40960 -P /var.skel swap /var
echo

The values given to mount_mfs are chosen carefully.

  • /dev holds a lot of very small files which consume a lot of inodes (relative to the data capacity needed) so it’s necessary to increase the number of inodes to one per 512 bytes of data space (default is one per 8192 bytes). If you run out of inodes when MAKEDEV is running, it’s very likely the system will not boot.
  • The size of the /dev file system has been set way higher than what is needed to store the actual device files, however it is necessary because the number of inodes available is proportional to the size of the file system.
Filesystem   1K-blocks Used Avail Capacity iused ifree %iused Mount
mfs:21550         1711   24  1602     1%    1522   524   74%  /dev
  • You can see above that only 1% of the data capacity has been used and 74% of the inodes.
  • The size of /var is set to give enough room for the skeleton files and syslogs. A 20MB size gives lots of headroom. The -P argument tells mount_mfs to populate the new RAM drive with the contents of the /var.skel directory.
  • The size of /tmp is pretty arbitrary; since there aren’t really any applications running on this system, very few files get written there. It’s also set to 20MB.

Important Symlinks

When the system restarts, any changes that were made to any files or directories on a RAM drive are lost. This impacts two directories of particular importance: /var/cron and /var/db/pkg. The solution is to store these directories on the actual flash drive itself. A symbolic link for each directory is created back into /etc in order to accomplish this.

root@mu:~# ls -l /var/cron /var/db/pkg
lrwxr-xr-x 1 root wheel  9 May 4 20:14 /var/cron -> /etc/cron
lrwxr-xr-x 1 root wheel 11 May 4 20:14 /var/db/pkg -> /etc/db/pkg

System Operation

Some notes on the firewall’s operation:

  • There is no swap space configured. Having swap on the compact flash card would not only greatly impact performance, but would decrease the lifetime of the card. There is enough physical RAM in the device that swapping shouldn’t be necessary anyways (knock wood).
  • The syslog daemon is setup to log in the usual /var/log location (which is on a RAM drive) as well as to a syslog server. Sending the logs to another server ensures the log archive is available even if the firewall reboots and the contents of the RAM drive are lost.
  • In order to work around the bug in the Broadcom BCM4401 chip mentioned above, the amount of RAM in the system is artificially limited via the /etc/boot.conf file with this entry: machine memory =1G
  • As a result of the issues in the BCM4401 chip, the OpenBSD project has disabled the bce(4) driver in the GENERIC kernel. The driver needs to be added back and the kernel recompiled in order to use those two NICs.

OpenBSD IPsec Tunnel Guide

This guide will explain how to setup a site-to-site IPsec tunnel (i.e., tunnel mode IPsec) between two OpenBSD gateways. Throughout this document there are example configs shown, some of which contain secret key data. DO NOT use these example keys! Create your own (as shown) and keep them private.

Table of Contents

The Tools

OpenBSD ships with all the tools needed to begin using IPsec. OpenBSD does not require a kernel recompile, software installtion, 3rd-party modules or anything else to get IPsec up and running.

The following is a list of OpenBSD’s IPsec tool set.

isakmpd(8)
The Internet Security Association and Key Management Protocol (ISAKMP) Daemon. This daemon is responsible for exchanging cryptographic keys and setting up security associations (SAs) with IPsec peers.
ipsecctl(8)
The ipsecctl(8) utility is used to create and delete the flows that determine which traffic is protected using IPsec. It’s also used as the primary method of configuring IPsec. It does this by reading its configuration file and then passing various commands to isakmpd(8) in order to establish the IPsec connection.
ipsec.conf(5)
The /etc/ipsec.conf file is the configuration file for the ipsecctl(8) utility. The flows and security associations that make up an IPsec connection are defined in this file using an easy-to-read syntax.
sasyncd(8)
The security association synchronization daemon. This daemon synchronizes IPsec SAs and flows between redundant gateways.
sasyncd.conf(5)
The /etc/sasyncd.conf file is the configuration file for sasyncd(8).

Terminology

Some of the terms and acronyms used when talking about IPsec are listed below.

Authentication Header (AH)
AH provides data integrity and origin authenticity of the payload. AH computes a hash using certain parts of the payload and sends the hash along with the packet payload. The receiver recomputes the hash and verifies it against the hash that it received in the packet to ensure that they match.
Encapsulating Security Payload (ESP)
ESP provides origin authenticity, integrity and confidentiality for the encapsulated IP packet. With ESP the payload of the IP packet is encrypted.
Internet Key Exchange (IKE)
IKE is the protocol that isakmpd(8) uses to setup Security Associations with IPsec peers.
Main Mode
When isakmpd(8) is negotiating with an IPsec peer using IKE, the first phase of the negotiation is called main mode. During main mode the two peers establish a secure channel which they then use to exchange the data needed to setup the security associations.
Quick Mode
When isakmpd(8) is negotiating with an IPsec peer using IKE, and after main mode has been completed, the two peers exchange the data necessary to setup the security associations. This exchange is called quick mode. Once the quick mode exchange is complete, data can flow across the IPsec connection.
Pre-Shared Key (PSK)
A secret key/password that is exchanged between two parties using some secure method. For example, generating a secret key and using SSH to upload they key to the IPsec gateways.
Security Association (SA)
A grouping of the security parameters that are used to protect the data sent over the IPsec connection. The parameters in the SA include key data and the encryption and hash algorithms. An SA controls traffic in one direction only. Two SAs are required on each gateway in order to achieve bidirectional IPsec traffic flow. SAs are established automatically by isakmpd(8) or can be created manually using ipsecctl(8).
Transport Mode
When an IPsec connection is run in transport mode only the payload of the original IP packet is encrypted. The packet header is left intact meaning the original destination address is used when routing the encrypted packet to its destination. Transport mode is used for host-to-host communication.
Tunnel Mode
When an IPsec connection is run in tunnel mode the entire original packet is encrypted. This encrypted packet is then encapsulated into a new IP packet. This new IP packet is routed to the remote IPsec peer. Tunnel mode is used for site-to-site (network-to-network) communication. Since the internal/site IP addresses are hidden inside the encrypted packet, these packets can be routed across public networks such as the Internet.

Building a Site-to-Site Tunnel

This scenario will describe building a site-to-site tunnel that’s used to connect two physically separate networks together over the public Internet. At each site is an OpenBSD gateway that will act as the endpoints for the IPsec connection. The end goal of this scenario is to enable full communication between the two networks.

Our sample topology:

 172.16.0.0/24     .1                  192.0.2.153
  (Network 1) --- em0[OpenBSD gateway 1]em1 -----+
                                                 |
                                            ((Internet))
                                                 |
  (Network 2) --- em0[OpenBSD gateway 2]em1 -----+
 192.168.15.0/24   .1                  192.0.123.85

Gather Information

Before anything can be configured we need to gather some basic information. We already have the necessary IP address information from the topology diagram above. We now need to define the IPsec parameters that will be used to build the connection.

Right off the bat we know that since we’re connecting two networks together using IPsec that we need to use tunnel mode. We also know that since we want to keep the transported data private (i.e., encrypted) that we need to use ESP as the transport protocol. Now we need to decide on the encryption and authentication types to use for main mode and quick mode. The easiest thing to do here is to not specify any encryption or authentication types and insead allow the default types to be used. Since both endpoints of the IPsec tunnel are OpenBSD and they both support the same set of encryption/authentication types, they will not have any difficulty negotiating with each other. For this exercise, a set of non-default types will be chosen in order to demonstrate how to configure them. The complete list of support enc/auth types is listed in the ipsec.conf(5) manpage in the “CRYPTO TRANSFORMS” section. For main mode we will use hmac-sha2-256 for authentication and blowfish for encryption. For quick mode we will use hmac-sha2-384 and blowfish.

Next we need to decide what method of authentication should be used between the gateways. To keep things simple, this example will use a pre-shared key (PSK). In order to ensure a strong PSK is used, we will use the openssl(1) utility to generate the key. We could just as easily pick our own PSK to use, however using OpenSSL ensures that 1) the PSK is random and not predictable and 2) that it’s sufficiently long enough that things like a brute-force attack are not feasible.

% openssl rand 20 | hexdump -e '20/1 "%02x"'
204931646879ae1265a7d0437a51cc08fe8d7484

This command has generated a random 20 byte (160 bit) string and formatted it in hexadecimal. This key should be treated with care and kept secret at all times. A secure method of transport should be used when uploading the key to the gateways such as using SSH or entering the key from the console.

Here is a summary of the information we’ve decided upon:

Gateway 1 IP
192.0.2.153
Gateway 2 IP
192.0.123.85
Network 1
172.16.0.0/24
Network 2
192.168.15.0/24
Tunnel/Transport Mode
Tunnel Mode
Transport Protocol
ESP
Main Mode enc/auth
hmac-sha2-256, blowfish
Quick Mode enc/auth
hmac-sha2-384, blowfish
Authentication method
Pre-shared key
Pre-shared key
204931646879ae1265a7d0437a51cc08fe8d7484

Creating the ipsec.conf(5) File

Now that we’ve defined all the configuration parameters we can begin configuring the tools. We’ll start with the /etc/ipsec.conf file. OpenBSD ships with a default ipsec.conf file that can be used as a starting point. We’re going to start with an empty file and configure everything from the ground up.

Since our intent is to have isakmpd(8) configure the flows and SAs automatically, we will define the IKE parameters in ipsec.conf. When ipsecctl(8) loads the config, it will instruct isakmpd to configure itself appropriately. The basic IKE configuration syntax is:

ike [mode] [encap] [tmode] \
 from src to dst \
 local localip peer remoteip \
 main auth algorithm enc algorithm \
 quick auth algorithm enc algorithm \
 psk key

These options are explained below.

mode
Defines whether isakmpd will attempt to initiate the IPsec connection or whether it will wait for a connection request from the remote peer. Possible values are:

  • active – Tells isakmpd to initiate the IPsec connection. This is the default.
  • passive – Tells isakmpd to wait for a connection request.
  • dynamic – Tells isakmpd to initiate the IPsec connection and to enable Dead Peer Detection. This mode should be used when the remote peer has a dynamic IP address.
encap
The transport protocol to use, either “esp” or “ah“.
tmode
Either “tunnel” or “transport” and indicates whether to enable tunnel mode or transport mode. The default is “tunnel“.
src
The IP address or subnet where the traffic to be protected is coming from. For a transport mode connection this will be the IP address of the local host. For a tunnel mode connection this will be the subnet and mask of the local network that should have its traffic sent through the IPsec tunnel.
dest
The IP address or subnet where the protected traffic is being sent to. For a transport mode connection this will be the IP address of the remote host. For a tunnel mode connection this will be the subnet and mask of the remote network that will be reachable through the IPsec tunnel.
localip
The IP address on the local machine where isakmpd should source the connection from. This option is generally not needed since isakmpd will choose the correct IP address automatically.
remoteip
The IP address of the remote IPsec gateway.
algorithm
The authentication or encryption algorithm to use during main mode and quick mode. The possible algorithms are documented in the ipsec.conf(5) man page under the “CRYPTO TRANSFORMS” section.
key
The pre-shared key.

Now that we have the syntax for creating the config file we can substitute our IPsec parameters from the table above into the file.

The ipsec.conf file on gateway 1:

ike esp from 172.16.0.0/24 to 192.168.15.0/24 \
 peer 192.0.123.85 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484

The ipsec.conf file on gateway 2:

ike esp from 192.168.15.0/24 to 172.16.0.0/24 \
 peer 192.0.2.153 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484

Note that even though we don’t specify the “active” or “tunnel” options they are the defaults and will be automatically applied by ipsecctl.

Starting isakmpd

Now that the ipsec.conf file is written we can tell ipsecctl(8) to load this config. When it does, it will instruct isakmpd to setup the IPsec connection based on the parameters we’ve specified. It’s then isakmpd’s job to create the SAs, setup the flows, and authenticate the peer.

To have isakmpd start when the system boots, edit the /etc/rc.conf.local file and add the following on a new line:

isakmpd_flags="-K"

The -K option to isakmpd tells it not to load a keynote(4) policy. This option is necessary when using ipsecctl to control isakmpd.

You can start isakmpd from the shell by passing it the same flag:

# isakmpd -K

Now load the ipsec.conf file using ipsecctl:

# ipsecctl -f /etc/ipsec.conf

At this point ipsecctl will have communicated with isakmpd to tell it to establish the IPsec connection. To have ipsecctl run at system boot, add the following to /etc/rc.conf.local:

ipsec=YES

At this point isakmpd should be running and the ipsec.conf file should be loaded on both gateways.

After a short time the IPsec connection will be established. You can view the flows and SAs using ipsecctl.

gw1# ipsecctl -s all
FLOWS:
flow esp in from 172.16.0.0/24 to 192.168.15.0/24 peer 192.0.123.85 \
  srcid 192.0.2.153/32 dstid 192.0.123.85/32 type use
flow esp out from 192.168.0.0/24 to 172.16.0.0/24 peer 192.0.123.85 \
  srcid 192.0.2.153/32 dstid 192.0.123.85/32 type use

SAD:
esp tunnel from 192.0.123.85 to 192.0.2.153 spi 0x... \ 
  auth hmac-sha2-384 enc blowfish \
  authkey 0x... \
  enckey 0x...
esp tunnel from 192.0.2.153 to 192.0.123.85 spi 0x... \
  auth hmac-sha2-384 enc blowfish \
  authkey 0x... \
  enckey 0x...

We can see here on gateway #1 two flows have been established, one in each direction. Underneath that are the entries in the security association database (SAD). Like the flows, there is an SA in each direction. The “0x…” indicates values that will be unique for each SA. If the IPsec connection is not established then the output above will only show the flows and not the SAs.

Assuming you can see the flows and SAs then the connection is ready to pass traffic.

Allowing IPsec Traffic Through pf(4)

Since most OpenBSD gateways will be running pf(4), it’s necessary to add some explicit rules to allow IPsec traffic to and from the remote gateway. Two types of traffic need to be permitted between gateways:

  1. UDP traffic on ports 500 and 4500. These are the ports that isakmpd uses to communicate.
  2. ESP or AH packets (depending which one the IPsec connection is using).

These rules can be used on gateway #1 to achieve this.

ipsec_peer = "192.0.123.85"
ext_if = "em1"

pass in on $ext_if proto udp from $ipsec_peer \
 to ($ext_if) port { 500 4500 }
pass out on $ext_if proto udp from $ext_if \
 to $ipsec_peer port { 500 4500 }

pass in on $ext_if proto esp from $ipsec_peer to ($ext_if)
pass out on $ext_if proto esp from $ext_if to $ipsec_peer

These rules would need to be integrated into the existing rules on gateway #1 and a similar set of rules would need to be integrated on gateway #2.

Filtering Traffic on the Tunnel

The previous section talked about filtering IPsec traffic to and from the gateway. Now we’ll look at how to filter user traffic that crosses the IPsec tunnel.

Filtering on enc0

Each OpenBSD gateway has a virtual enc(4) interface. This interface receives an unencrypted copy of all traffic sent and received on all IPsec connections. By using pf(4) to filter traffic on this interface we can control the types of traffic permitted on the tunnel.

The simplest filter is of course to allow all traffic inbound and outbound. This can be achieved using the “set skip” option.

set skip on enc0

Of course much more fine-grained control can be applied as well. Let’s assume in our sample topology that we only want to allow users on network #1 to be able to access a specific web server located on network #2. Users on network #2 should not be able to access any resources on network #1. By applying some filter rules to the enc0 interface on gateway #1 we can achieve this policy:

ext_if = "em1"
int_if = "em0"
ipsec_peer = "192.0.123.85"
www_server = "192.168.15.5"
www_ports = "{ 80 443 }"

block on enc0

pass in on enc0 proto ipencap from $ipsec_peer to ($ext_if) \
 keep state (if-bound)

pass out on enc0 proto tcp from $int_if:network to $www_server \
 port $www_ports keep state (if-bound)

Two items need special attention here.

  • Passing ipencap traffic: Because of how the OpenBSD IP stack works, incoming tunnel mode traffic must be explicity permitted by passing packets of protocol type “ipencap”. A corresponding outbound rule is not necessary because the stack handles outbound tunnel mode packets slightly differently. Care should be taken to only allow ipencap traffic from the specific hosts that we have IPsec connections with. If the remote IPsec peer has a dynamic IP address this obviously won’t be possible and the above filter rule would have to be modified to not filter on source IP address. If the IPsec connection is using transport mode then this rule is not needed.
  • Interface-bound state entries: Filter rules that permit user traffic over the tunnel should specify the “if-bound” state option. Without this option, if the IPsec connection is down (for example, if the remote gateway is offline) then users’ traffic will be routed using the normal routing table which would result in this traffic being sent over the public Internet. Using “if-bound” ensures that this traffic will only be permitted across the enc0 interface and therefore only across the IPsec tunnel. Care should also be taken when writing filter rules that permit regular user traffic to and from the Internet. Ensure that these rules do not inadvertantly allow private traffic to leak to the Internet when the IPsec connection is down.

Filtering using pf Tags

A pf tag is an internal identifier that can be attached to a packet as it enters an OpenBSD host. The tag can then be used as filter criteria by pf. See the Packet Tagging section of the PF User’s Guide for more details on tagging. The isakmpd daemon is capable of applying pf tags to incoming IPsec packets.

In order to apply a tag to an IPsec connection it must specified in the ipsec.conf file using the “tag” keyword. Taking our existing ipsec.conf file from gateway #2, we simply add the tag to the end:

ike esp from 192.168.15.0/24 to 172.16.0.0/24 \
 peer 192.0.2.153 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484 \
 tag net1-to-net2

In this case the arbitrary string “net1-to-net2” was chosen as the tag. All incoming IPsec packets belonging to this connection will be tagged with “net1-to-net2”. We can now incorporate this tag into the pf filter rules on gateway #2. Since we know from the previous section that we only want to allow users on network #1 to access a specific web server on network #2, we can apply this policy on gateway #2 by using the pf tag:

ext_if = "em1"
int_if = "em0"
ipsec_peer = "192.0.2.153"
www_server = "192.168.15.5"
www_ports = "{ 80 443 }"

block on enc0

pass in on enc0 proto ipencap from $ipsec_peer to ($ext_if) \
 keep state (if-bound)

pass in on enc0 keep state (if-bound) tagged net1-to-net2

pass out on $int_if proto tcp to $www_server port $www_ports \
 keep state (if-bound) tagged net1-to-net2

This example ruleset allows users on network #1 to access the web server on network #2 by matching on the pf tag that’s being applied to packets by isakmpd. An additional rule is added on $int_if to demonstrate how filtering on the tag and on IP address and port can be combined.

Filtering Traffic Using IKE Policy

The last way of filtering traffic is by using the IKE policy to limit what kind of traffic will be encapsulated and sent across the tunnel. Within the ipsec.conf file it’s possible to define the IP protocol and TCP/UDP ports to allow traffic to and from. Continuing with our example of only allowing network #1 to reach a web server on network #2, we could adjust the ipsec.conf file on gateway #1 to read:

ike esp proto tcp from 172.16.0.0/24 to 192.168.15.0/24 port 80 \
 peer 192.0.123.85 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484

When this ipsec.conf is loaded by ipsecctl, it will setup a flow between 172.16.0.0/24 and 192.168.15.0/24 for TCP port 80 only. Similar changes should be made on gateway #2.

There’s one thing we’re missing now though. In the previous filtering methods we allowed port 443 traffic to reach the web server in addition to port 80. How do we allow more than one port (or protocol) using this method? The answer is create a second IKE policy that specifes port 443:

ike esp proto tcp from 172.16.0.0/24 to 192.168.15.0/24 port 443 \
 peer 192.0.123.85 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484

A matching policy needs to be created on gateway #2.

Adding Redundancy

OpenBSD has built-in support for deploying redundant IPsec gateways. Synchronization of SAs and flows is handled by the sasyncd(8) daemon. This daemon is most often used in conjunction with carp(4). By enabling CARP on the redundant gateways they able to share a common external IP address. Configuring all IPsec connections to use this shared IP address allows the connections to failover between the redundant gateways.

We’ll modify our example topology to include a set of two redundant gateways connected to network #1.

 172.16.0.0/24     .2            192.0.2.154
  (Network 1) --- em0[OpenBSD gateway 1a]em1 --------+
           |    .1 carp1          carp2 192.0.2.153  |
           +----- em0[OpenBSD gateway 1b]em1 -+      |
                   .3            192.0.2.155  |      |
                                              |      |
                                            ((Internet))
                                                 |
  (Network 2) --- em0[OpenBSD gateway 2]em1 -----+
 192.168.15.0/24   .1                  192.0.123.85

Network #1 now has two gateways: gateway #1a and #1b. These gateways are running CARP on their internal and external interfaces.

  • Internal: carp1, 172.16.0.1
  • External: carp2, 192.0.2.153

The general operation and configuration of CARP is covered in FAQ6: The Common Address Redundancy Protocol and in the PF User’s Guide Firewall Redundancy with CARP and pfsync.

The ipsec.conf file used on gateway #1a and #1b is mostly the same as in the earlier sections. The only thing that needs changing is to add the “local” keyword which tells isakmpd which IP address to bind to. Without this option, isakmpd would automatically use the IP address assigned to the physical em0 interface.

ike esp from 172.16.0.0/24 to 192.168.15.0/24 \
 local 192.0.2.153 peer 192.0.123.85 \
 main auth hmac-sha2-256 enc blowfish \
 quick auth hmac-sha2-384 enc blowfish \
 psk 204931646879ae1265a7d0437a51cc08fe8d7484

By telling isakmpd to use the shared CARP IP address we ensure that traffic will continue to flow when the gateways failover. Likewise on gateway #2, the ipsec.conf file should be changed so that the “peer” IP is set to the external CARP IP, 192.0.2.153.

The next bit of redundancy needed is for the IPsec SAs and flows. This is handled by sasyncd(8). We’ll use a very basic sasyncd.conf file with these options:

interface
The name of the CARP interface to track. When this interface is in “master” state then sasyncd will know the gateway is the current master.
peer
The IP address of the other gateway. This is the address that synchronization messages will be sent to.
sharedkey
A pre-shared key used for encrypting updates sent between gateways.

The sasyncd.conf file on gateway #1a looks like:

# track the external carp interface
interface carp2
# address of gateway #1b
peer 172.16.0.3
# 256 bit shared key
sharedkey 0xda23169c9f49bb1fcdfab7d556588975433f831b3723d273a06e9d780f2cc

To start sasyncd on system boot, add a line to /etc/rc.conf.local:

sasyncd_flags=""

Check your pf.conf ruleset to ensure that traffic between the internal interfaces of gateway #1a and #1b on TCP port 500 is permitted. This is the port that sasyncd will use.

One last thing to do and that is if isakmpd is already running on gateway #1a or #1b, kill it and restart it by adding the -S flag to its startup options. This option tells isakmpd not to delete SAs when it’s killed. The system startup scripts will automatically add the -S option when sasyncd is enabled.

At this point the tunnel should be up between the active gateway and gateway #2. Traffic should be flowing as before. On gateway #1b (the standby gateway), you should be able to see the IPsec SAs and flows which are being synchronized from gateway #1a using the ipsecctl -s all command.

Once the SAs are synchronized it should be possible to fail over between gateway #1a and #1b without disruption to the IPsec connection.

Troubleshooting

Some troubleshooting steps in case things aren’t working correctly.

Startup Order is Important

When OpenBSD boots, the rc(8) startup script runs the various IPsec tools in the correct order to ensure that everything works as it should (assuming of course that all the correct tools are enabled in /etc/rc.conf.local). If you’re starting the tools by hand from the command line, do so in this order:

  1. Run isakmpd with the -K and -S flags
  2. Load the /etc/ipsec.conf file with ipsecctl
  3. Run sasyncd

OpenBSD OpenBGPD Notes

OpenBGPD is a free, open-source implementation of the Border Gateway Protocol Version 4. It was created and is maintained by the OpenBSD project.

The notes here apply to OpenBGPD as found in OpenBSD 4.0 and higher.

Path Selection Process

OpenBGPD will only ever install one route in the route table for a particular destination network (prefix). If OpenBGPD receives information about that prefix from more than one peer, a decision must be made on which one to use. The prefixes received will be evaluated against each other if the follow criteria matches:

  • Prefix length is the same
  • Both routes are for the same destination network
  • The NEXT_HOP is reachable

OpenBGPD uses the following process to determine the “best” route:

  1. Local Preference. Higher local preference is preferred.
  2. AS_PATH length. Shorter AS_PATH is preferred.
  3. Origin code. Lower origin code is preferred. (IGP > EGP > Incomplete)
  4. Multi Exit Discriminator. Lowest MED is preferred. Note that MEDs are only compared if the two routes where announced to us from the same AS. (The MED can be used even if the peers are in different ASes using rde med compare always)
  5. Prefer eBGP learned routes over iBGP learned routes.
  6. Weight. This is administratively set using the weight keyword in the bgpd config file. Note that by default, the weights are equal.
  7. Route age. The older route (i.e., the more stable route) is preferred. Note that this step is only performed when the rde route-age option is set to evaluate. By default it is set to ignore.
  8. BGP ID of the peer that announced the route to us. Lowest BGP ID is preferred.
  9. Lowest peer address. This is the final, tie-breaking rule. The BGP peer that has the lowest IP address wins.

If at any step in the process a route is found to be more preferred than the other, the process is aborted and that route is taken as the “best” route to the destination network.

UPDATE (Aug 2012): The bgpd path selection process is now documented in the bgpd(8) man page.

Next Hop Resolution

When looking for and validating next-hop addresses, OpenBGPD will only consider static routes and routes added by other dynamic routing protocols. BGP-learned routes and the default route are not considered. This behavior can be overridden using the nexthop qualify via configuration option.

BGP Path Attributes

The following BGP path attributes can be controlled via bgpd.conf.

LOCAL_PREF
Default value for routes received from iBGP peers is 100. Default value for routes recieved from eBGP peers is 0. LOCAL_PREF is never advertised to eBGP peers.
MULTI_EXIT_DISC (MED)
The MED is only advertised to an eBGP peer if it’s being sourced from the local router. The MED is always advertised to iBGP peers (if it’s present for a given prefix).
Communities
Communities are advertised to all peers (iBGP and eBGP).
Next-hop
By default the next-hop is set to whatever the peer advertises as the next-hop. The next-hop can be set to an alternate address or can be used to create null routes by using the blackhole or reject target.
Weight
Weight is a local attribute and is never redistributed to peers. It is used as tie-breaking criteria when comparing two equal routes.

Refer to the bgpd.conf(5) man page for details on how to manipulate the path attributes.

References

OpenBSD CARP Notes

CARP is the Common Address Redundancy Protocol. It’s a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project.

The notes here apply to OpenBSD 5.0 and higher.

Protocol Information

Virtual MAC Address
The virtual MAC is in the format 00-00-5e-00-01-XX where the last octet is filled in by the CARP vhid.
IP Protocol
CARP uses IP protocol number 112 (0x70).
Multicast Advertisements
CARP advertisements are multicast to the 224.0.0.18 or FF02::12 multicast groups when using IPv4 and IPv6, respectively.
TTL/Hop Limit
CARP packets are always sent with a TTL/HLIM of 255 so that CARP packets that have crossed a subnet boundary (i.e., have been passed on by a router) can be recognized and dropped.

Timers

The host that advertises the most frequently will become the master for the CARP group. The timer values configured on each host are sent as part of the CARP advertisements so that all hosts can make an accurate decision as to which host should be master.

Advertisement Interval
This is the base interval at which CARP advertisements will be sent. The default is 1 second and is configured with the advbase keyword. In OpenBSD 5.0 onwards, a value of 0 may be specified if sub-second failover between nodes is required.
Advertisement Skew
This value is used to skew the advertisement interval of a host in order to make it more or less preferred in becoming master. A higher skew value causes a host to send CARP advertisements a fraction of a second slower than hosts with a lower value thereby making it less preferred as the master. The valid range is 0 to 254 with the default being 0. Configure skew with the advskew keyword.

When advbase is set to 0 the skew value alone is used to calculate how often advertisements are sent (the advertisement window) using this formula:

window in microseconds = advskew * 1000000 / 256

Eg: 100 * 1000000 / 256 = 390625 ┬Ás

As shown, a skew value of 100 (and interval value of 0) results in an advertisement window of 0.39 seconds.

Failover Timer
If a backup CARP host doesn’t see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows to delay before assuming the master is down is hard-coded into CARP and is not tunable.

In the event that two or more hosts have the same timer values configured, the following behavior results:

  • If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.
  • If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.

The Demotion Counter

Another metric used in determining which host becomes master is the demotion counter. The demotion counter is a value advertised by CARP hosts that announce how “ready” a host is to take on the role of master.

The values used to calculate the demotion counter are stored in dynamic interface groups. By default, each CARP interface is a member of the carp interface group.

carp100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
 groups: carp
 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255

The demotion value is viewed and set using ifconfig(8).

# ifconfig -g carp
carp: carp demote count 0
# ifconfig -g carp carpdemote 100
# ifconfig -g carp
carp: carp demote count 100

Here the demotion value of the carp interface group is set to 100.

When CARP advertises on the network, it takes the sum of the demotion values of all interface groups that the CARP interface is a member of and advertises that as the demotion counter. Hosts with higher values are less preferred to become master for that particular CARP group.

The demotion counter makes it possible to failover selected CARP groups rather than the “all or nothing” approach used with preemption.

Interface States

INIT
All CARP interfaces start in this state. Also, when a CARP interface is admin down it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled.
BACKUP
The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.
MASTER
The host is forwarding traffic directed to the virtual/group IP address. The host is also sending advertisements once per advertisement window that announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.

Note that changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.

Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host will ever be in MASTER state.

CARP Bugs

Preempt Failover Race
In the following scenario, a race occurs: Two firewalls each connected to switches using two separate physical interfaces on the firewall. The firewalls have preempt enabled; fw01 is master, fw02 is backup on all CARP groups. One of the switches goes away causing the interfaces on the firewalls to go down. Since both firewalls have preempt enabled, they up their advskew to 240 on all remaining CARP groups. Here’s the race: Who becomes master now? There was an interesting discussion about this on the OpenBSD pf mailing list here: http://marc.theaimsgroup.com/?l=openbsd-pf&m=113881646826219&w=2. Steven S commented that his workaround is to only set preempt on the master firewall and to disable it on the other firewall(s). There is no permanent solution to this issue currently.

References

OpenBSD SNMP MIBs

The following SNMP MIBs and the accompanying code that extend the Net-SNMP daemon allow administrators to query information from various OpenBSD subsystems. Currently, stats can be queried from:

  • Packet Filter
  • The kernel sensors framework
  • Common Address Redundancy Protocol (CARP)

These MIBs are being integrated into OpenBSD’s own snmpd. OpenBSD 5.1 has the kernel sensor and CARP MIBs. OpenBSD 5.1-current has and the future 5.2 release will have the pf MIB. See this post for a bit more detail.

Table of Contents

SNMP Introduction

SNMP is the Simple Network Management Protocol. It’s used to manage nodes (routers, switches, servers, etc) on an IP network. SNMP allows an administrator to query the current status of a node, collect data for analysis and historical reasons, and to make configuration changes to the node. SNMP also has provisions for a node to send an alarm or a “trap” to a central monitoring station to alert the administrator to possible trouble.

When the node being monitored is a router or firewall, some of the more commonly monitored objects include:

  • Interface bytes per second
  • Interface packets per second
  • Bytes passed/blocked
  • Packets passed/blocked
  • CPU usage
  • Memory usage
  • State table size

The list of objects that can be monitored by SNMP is stored in a Management Information Base, or MIB. The MIB defines the name of the object, its data type (integer, string, etc) and the object’s location within the MIB heirarchy.

Packet Filter MIB

The PF MIB allows for the querying of objects relating to OpenBSD’s Packet Filter firewall software. The following information can be queried from the MIB:

  • General info (is PF enabled, the runtime, the hostid)
  • Packet counters
  • State table counters
  • Log interface counters
  • Source track counters
  • Memory limits
  • Protocol timeouts
  • Interface stats (number of rules, number of states, bytes passed/blocked, packets passed/blocked, etc)
  • Table stats (number of addresses, evaluations, bytes passed/blocked, packets passed/blocked, etc)
  • Table content stats (bytes passed/blocked and packets passed/blocked for each address/network within a table)
  • Filter rule label counters
    • This feature was contributed by Sven Ingebright Ulland <email hidden; JavaScript is required>
  • pfsync(4) protocol counters

The limitations of the MIB:

The MIB tree can be viewed here: PF MIB tree
A sample walk of the MIB can be seen here: PF MIB walk

Kernel Sensors MIB

The Kernel Sensors MIB allows for querying information from the OpenBSD kernel sensors. The kernel sensors provide data from various environmental sensors, disk drives, etc.

The MIB reports data in much the same way as the “sysctl hw.sensors” command does.

The MIB tree can be viewed here: Sensors MIB tree
A sample walk of the MIB can be seen here: Sensors MIB walk

CARP MIB

The CARP MIB allows for querying the status of carp(4) interfaces. It can be used to determine how many CARP groups are configured on a node, what their state is, and the various operational settings of the CARP group, including advskew, advbase and the carpdev.

The limitations of the MIB:

  • The value of the CARP demote counter associated with interface groups is not handled.
  • The IP address(es) assigned to the CARP interface is not returned. This can be queried in the ifTable.
  • When doing load balancing via CARP, the “carpnodes” are not shown via the MIB.

The MIB tree can be viewed here: CARP MIB tree
A sample walk of the MIB can be seen here: CARP MIB walk

Download

The MIBs are available as a patch to the OpenBSD net-snmp port.

Earlier releases only included the PF MIB. These are listed below.

The pf-mib39 release is backwards compatible with OpenBSD 3.8, however the diff will not apply cleanly to the OpenBSD 3.8 net-snmp port. You will have to resolve any failures by hand.

The OLD, ucd-snmp patch is still available here: pf-mib-ucd.diff. This diff works with ucd-snmp 4.2.6 on OpenBSD 3.5 and 3.6.

Installation

Perform these steps to install the MIB:

  1. Retrieve the net-snmp port from the OpenBSD ports system
  2. Extract the patch tarball
  3. Apply the included patch
  4. Build and install the port

Example:

# cd /usr/ports/net/net-snmp
# ftp http://www.packetmischief.ca/files/openbsd/snmp/obsd-mibsXX.tar
# tar xf obsd-mibsXX.tar
# patch < obsd-mibs.diff
# make install

The MIBs are now installed as part of the net-snmp port.

Usage

The path to the MIBs is .1.3.6.1.4.1.64512 which translates to iso.org.dod.internet.private.enterprises.openBSD. 64512 is a somewhat arbitrarily chosen number that is not currently assigned.

To see what objects are available in the MIB you can either walk the MIB using an SNMP client…

# snmpwalk -v2c -c community host OPENBSD-PF-MIB::pfMIBObjects
# snmpwalk -v2c -c community host OPENBSD-SENSORS-MIB::sensorsMIBObjects
# snmwwalk -v2c -c community host OPENBSD-CARP-MIB::carpMIBObjects

… or you can view the tree:

# snmptranslate -Tp -OS OPENBSD-PF-MIB::pfMIBObjects
# snmptranslate -Tp -OS OPENBSD-SENSORS-MIB::sensorsMIBObjects
# snmptranslate -Tp -OS OPENBSD-CARP-MIB::carpMIBObjects

To query an object in the MIB, find the path to the object you want and append it to .enterprises.openBSD. For example, if you wanted to query the running object, you would determine its path to be pfMIBObjects.info.running.0. Note the 0 (zero) on the end; it is important, so don’t omit it. Next you need to tell your snmp client to load the MIB so that it’s able to translate the above path to the numeric path. The sequence looks like this:

# export MIBS="+/usr/local/share/snmp/mibs/OPENBSD-PF-MIB.txt"
# snmpget -v2c -c community host \
.enterprises.openBSD.pfMIBObjects.info.running.0
enterprises.openBSD.pfMIBObjects.info.running.0 = true(1)

The output from the snmpget is shown in bold.

Example Graphs

Below are a couple of graphs from a Cacti server that is monitoring some OpenBSD firewalls that have these MIBs installed.

 

Third-Party Tools

Below is a list of tools that others have written to take advantage of the MIBs available here. If you want something added here, please contact me.

  • Nagios check_pf_carp. Generates alarms for CARP hosts that are not in MASTER state. By Brian A. Seklecki

Credits

The following people have contributed by writing code, reporting bugs, offering suggestions, or sending alchohol :-)

  • Sven Ingebrigt Ulland (PF-MIB: labels support)
  • Brian A. Seklecki
  • Bryan Kaplan
  • Bill Cameron
  • Clint Byrum

Thank you for your contributions.

Related Blog Posts

References