Notes About OpenBSD's CARP
Introduction
CARP is the Common Address Redundancy Protocol. It's a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project.
The notes here apply to OpenBSD 4.0.
Protocol Information
- Virtual MAC Address
- The virtual MAC is in the format 00-00-5e-00-01-XX where the last octet is filled in by the CARP vhid.
- IP Protocol
- CARP uses IP protocol number 112 (0x70).
- Multicast Advertisements
- CARP advertisements are multicast to the 224.0.0.18 or FF02::12 multicast groups when using IPv4 and IPv6, respectively.
- TTL/Hop Limit
- CARP packets are always sent with a TTL/HLIM of 255 so that CARP packets that have crossed a subnet boundary (i.e., have been passed on by a router) can be recognized and dropped.
Timers
The host that advertises the most frequently will become the master for the CARP group. The timer values configured on each host are sent as part of the CARP advertisements so that all other hosts can make a determinate decision as to which host will become the master.
- Advertisement Interval
- This is the base interval at which CARP hellos will be sent. The default is 1 second and is configured with the advbase keyword.
- Advertisement Skew
- This value is used to skew the advertisement interval in order to make the host more or less preferred in becoming master. The valid range is 0 to 254, with lower values making the host more preferred to be master. The default is 0 and is configured with the advskew keyword.
The advertisement window is calculated by taking the skew, dividing it by 256 and adding it to the interval. The CARP host will send a hello every "window" seconds.
- Failover Timer
- If a backup CARP host doesn't see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows to delay before assuming the master is down is hard-coded into CARP and is not tuneable.
In the event that two or more hosts have the same timer values configured, the following behavior results:
- If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.
- If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.
The Demotion Counter
Another metric used in determining which host becomes master is the demotion counter. The demotion counter is a value advertised by CARP hosts that announce how "ready" a host is to take on the role of master.
The values used to calculate the demotion counter are stored in dynamic interface groups. By default, each CARP interface is a member of the carp interface group.
carp100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
groups: carp
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
The demotion value is viewed and set using ifconfig(8).
# ifconfig -g carp
carp: carp demote count 0
# ifconfig -g carp carpdemote 100
# ifconfig -g carp
carp: carp demote count 100
Here the demotion value of the carp interface group is set to 100.
When CARP advertises on the network, it takes the sum of the demotion values of all interface groups that the CARP interface is a member of and advertises that as the demotion counter. Hosts with higher values are less preferred to become master for that particular CARP group.
The demotion counter makes it possible to failover selected CARP groups rather than the "all or nothing" approach used with preemption.
Interface States
- INIT
- All CARP interfaces start in this state. Also, when a CARP interface is admin down it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled.
- BACKUP
- The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.
- MASTER
- The host is forwarding traffic directed to the virtual/group IP address. The host is also sending advertisements once per advertisement window that announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.
Note that changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.
Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host will ever be in MASTER state.
CARP Bugs
- IPv4 ARP Balance
- Matt Bradford debugged an issue with IPv4 ARP
Balance on OpenBSD 3.6 and up. In short, a CARP host would modulate the
CARP virtual IP address instead of the source address of the incoming
packet. The end result is that traffic was routed by only one of the
CARP hosts and not balanced at all.
Matt's website explains the issue.
http://www.isi.qut.edu.au/people/mbradfor/openbsd-carp-arpbalance.html.
NOTE: that his page doesn't mention this, but this issue is fixed in OpenBSD 3.9 and up (src/sys/netinet/ip_carp.c v1.112). - Preempt Failover Race
-
In the following scenario, a race occurs: Two firewalls each connected
to switches using two separate phyiscal interfaces on the firewall.
The firewalls have preempt enabled; fw01 is master, fw02 is backup on
all CARP groups.
One of the switches goes away causing the interfaces on the firewalls to
go down. Since both firewalls have preempt enabled, they up their
advskew to 240 on all remaining CARP groups. Here's the race: Who
becomes master now?
There was an interesting discussion about this on the OpenBSD pf mailing list here: http://marc.theaimsgroup.com/?l=openbsd-pf&m=113881646826219&w=2. Steven S commented that his workaround is to only set preempt on the master firewall and to disable it on the other firewall(s). There is no permanent solution to this issue currently.
References