OpenBSD CARP Notes

CARP is the Common Address Redundancy Protocol. It’s a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project.

The notes here apply to OpenBSD 5.0 and higher.

Protocol Information

Virtual MAC Address
The virtual MAC is in the format 00-00-5e-00-01-XX where the last octet is filled in by the CARP vhid.
IP Protocol
CARP uses IP protocol number 112 (0x70).
Multicast Advertisements
CARP advertisements are multicast to the 224.0.0.18 or FF02::12 multicast groups when using IPv4 and IPv6, respectively.
TTL/Hop Limit
CARP packets are always sent with a TTL/HLIM of 255 so that CARP packets that have crossed a subnet boundary (i.e., have been passed on by a router) can be recognized and dropped.

Timers

The host that advertises the most frequently will become the master for the CARP group. The timer values configured on each host are sent as part of the CARP advertisements so that all hosts can make an accurate decision as to which host should be master.

Advertisement Interval
This is the base interval at which CARP advertisements will be sent. The default is 1 second and is configured with the advbase keyword. In OpenBSD 5.0 onwards, a value of 0 may be specified if sub-second failover between nodes is required.
Advertisement Skew
This value is used to skew the advertisement interval of a host in order to make it more or less preferred in becoming master. A higher skew value causes a host to send CARP advertisements a fraction of a second slower than hosts with a lower value thereby making it less preferred as the master. The valid range is 0 to 254 with the default being 0. Configure skew with the advskew keyword.

When advbase is set to 0 the skew value alone is used to calculate how often advertisements are sent (the advertisement window) using this formula:

window in microseconds = advskew * 1000000 / 256

Eg: 100 * 1000000 / 256 = 390625 µs

As shown, a skew value of 100 (and interval value of 0) results in an advertisement window of 0.39 seconds.

Failover Timer
If a backup CARP host doesn’t see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows to delay before assuming the master is down is hard-coded into CARP and is not tunable.

In the event that two or more hosts have the same timer values configured, the following behavior results:

  • If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.
  • If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.

The Demotion Counter

Another metric used in determining which host becomes master is the demotion counter. The demotion counter is a value advertised by CARP hosts that announce how “ready” a host is to take on the role of master.

The values used to calculate the demotion counter are stored in dynamic interface groups. By default, each CARP interface is a member of the carp interface group.

carp100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
 groups: carp
 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255

The demotion value is viewed and set using ifconfig(8).

# ifconfig -g carp
carp: carp demote count 0
# ifconfig -g carp carpdemote 100
# ifconfig -g carp
carp: carp demote count 100

Here the demotion value of the carp interface group is set to 100.

When CARP advertises on the network, it takes the sum of the demotion values of all interface groups that the CARP interface is a member of and advertises that as the demotion counter. Hosts with higher values are less preferred to become master for that particular CARP group.

The demotion counter makes it possible to failover selected CARP groups rather than the “all or nothing” approach used with preemption.

Interface States

INIT
All CARP interfaces start in this state. Also, when a CARP interface is admin down it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled.
BACKUP
The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.
MASTER
The host is forwarding traffic directed to the virtual/group IP address. The host is also sending advertisements once per advertisement window that announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.

Note that changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.

Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host will ever be in MASTER state.

CARP Bugs

Preempt Failover Race
In the following scenario, a race occurs: Two firewalls each connected to switches using two separate physical interfaces on the firewall. The firewalls have preempt enabled; fw01 is master, fw02 is backup on all CARP groups. One of the switches goes away causing the interfaces on the firewalls to go down. Since both firewalls have preempt enabled, they up their advskew to 240 on all remaining CARP groups. Here’s the race: Who becomes master now? There was an interesting discussion about this on the OpenBSD pf mailing list here: http://marc.theaimsgroup.com/?l=openbsd-pf&m=113881646826219&w=2. Steven S commented that his workaround is to only set preempt on the master firewall and to disable it on the other firewall(s). There is no permanent solution to this issue currently.

References

Leave a Reply

Your email address will not be published. Required fields are marked *

Would you like to subscribe to email notification of new comments? You can also subscribe without commenting.

Networking. Unix. Cyber Security. Code. Protocols. System Design.