Virtualizing the OpenBSD Routing Table


The OpenBSD routing table can be carved into multiple virtual routing tables allowing complete logical separation of attached networks. This article gives a brief overview of rtables and explains how to successfully leak traffic between virtual routing domains.

The ability to virtualize the routing table in OpenBSD first appeared in version 4.6. Since then the functionality has matured nicely with support for virtual routing tables now present in userland tools such as dhclient(8) and dhcpd(8) and in the routing protocol daemons ripd(8), ospfd(8), and bgpd(8). Kernel side, pf(4) has been extended to handle filtering of packets based on the routing table they came in on as well as being able to move packets between routing tables. This article will concentrate on the latter with examples of how to setup separate routing tables and leak traffic between them successfully.

Using separate routing tables is similar to using VRFs in Cisco IOS or routing instances in Juniper’s JUNOS. Multiple routing tables are created each of which contain their own forwarding and ARP information. In OpenBSD, each routing table is called an “rtable”. Network interfaces can be bound to an rtable which causes traffic going through the interface to be forwarded based on the information present in that rtable. When one or more interfaces are bound to an rtable, the rtable and all of the interfaces bound to it are called a routing domain, or “rdomain”.

Basic Configuration

Creating an rtable is done using route(8) with the -T argument.

# route -T 1 add

This creates rtable 1 if it doesn’t already exist and adds a default route to it.

Interfaces are bound to an rtable using ifconfig(8) with the “rdomain” keyword.

# ifconfig vic1 rdomain 1

This binds the vic1 interface to rtable 1.

To execute a command within a non-default rtable, use the route(8) command with the exec keyword.

$ route -T 1 exec telnet

This executes the telnet command within rtable 1. Certain commands such as ping(8) and arp(8) have their own command line arguments that will place them into an rtable (the -V argument in this case).

Setting up rdomains

By default, all interfaces on an OpenBSD host belong to rdomain 0. Traffic can flow freely between all interfaces (assuming the pf(4) ruleset allows it) without any special handling. Similarly, traffic can flow between all interfaces in the same non-default routing domain without any special handling (again, as long as the pf(4) ruleset passes this traffic).

Two OpenBSD Routing Domains

In this network, Host 1 and Host 2 both belong to rdomain 1. Routing domain 1 has routes to the 192.168.1/24 and 172.16.0/24 networks because they are directly attached so traffic between the two is forwarded without any special consideration. Host 1 and 2 cannot talk to Host 0 because Host 0 is connected to a separate routing domain.

As shown in the picture, pf(4) is used to connect routing domains. This is really powerful because pf(4) allows for very fine-grained packet matching which means you can be as specific or broad as you want when it comes to what traffic you want to pass between rdomains. Sending traffic between rdomains is done by using the rtable keyword in pf.conf.

pass in on vic1 to rtable 0
pass out on vic0

This is the basic ruleset needed to allow Host 1 to initiate a connection to Host 0.

The rtable must be specified on the rule that matches traffic inbound to the OpenBSD router. As stated in the pf.conf(5) man page, the resulting route lookup will only work correctly if the rtable is specified on the inbound rule. This ruleset is not enough for traffic to flow bidirectionally. We also have to look at the routing entries within the source and destination routing domains.

The source routing domain, in this case rdomain 1, is easy. pf(4) will magically handle taking the packets out of rdomain 1 and sending them to rdomain 0 — we do not need a route for 172.16.2/24 in rtable 1. Reverse traffic is different. Routing domain 0 requires a route be present for 192.168.1/24. The next-hop for this route isn’t really important, what’s important is that it’s present in the rtable. If a route isn’t present, then the route lookup will fail before pf(4) has a chance to move the packet into rdomain 1 and the return traffic will be dropped. Note that the route doesn’t have to be exactly 192.168.1/24, it could be 192.168/16 or even — the important part is that there is some kind of route in rtable 0 that will match the network in rdomain 1.

# route -T 0 add 192.168.1/24 -iface

This is kind of a cheat. It creates a route for 192.168.1/24 as a connected route on the rdomain 0 interface. Obviously this isn’t correct, but it doesn’t really matter. It achieves the goal of getting a route into rtable 0. Host 1 can now successfully talk to Host 0.

An alternative to creating a “connected” route is to set the next-hop of the 192.168.1/24 route to the loopback IP.

# route -T 0 add 192.168.1/24

The loopback interface provides a really convenient place to point your reverse path routes.

The caveat with this is that pf(4) must be active on the loopback interface you create. The default pf.conf ruleset contains “set skip on lo” which disables pf(4) on each loopback interface and will result in return traffic being dropped. Be sure that your loopback isn’t being “skipped”.

The same idea works between two non-default routing domains.

Three OpenBSD Routing Domains

Creating a loopback interface in rdomain 2 so that Host 1 can talk to Host 2 would look like:

# ifconfig lo2 rdomain 2
# route -T 2 add 192.168.1/24

Since lo2 is created inside rdomain 2, the IP address assigned to it doesn’t conflict with lo0 in rdomain 0.

Another caveat with the pf(4) ruleset is that the states that get created by the rule that specifies the rtable must be “floating”.

If you’ve changed the “state-policy” option in your pf.conf from the default of “floating” then you must use the “floating” keyword in your inbound rule.

set state-policy if-bound
pass in on vic1 to rtable 0 keep state (floating)
pass out on vic0

All of the above guidance also applies if you’re doing NAT on the outbound interface.

pass in on vic1 to rtable 0
pass out on vic0 nat-to vic0

This ruleset would hide the 192.168.1/24 network from hosts in rdomain 0 by translating the source 192.168.1.x IP to the IP address on the vic0 interface. This might be necessary if there’s already a 192.168.1 network in rdomain 0. Even though you’re doing NAT, you still need a route in rdomain 0 that points back to the real source network (192.168.1/24) in rdomain 1.

Sample Use Cases

Routing domains can be used to isolate a test/dev network from production.

Two OpenBSD Routing Domains

In the sample network from earlier, rdomain 0 could be the production network with production servers and the users connected to it. Routing domain 1 could be a test network where applications and systems are put through testing before being moved into rdomain 0. In order to prevent the test systems from possibly affecting the production systems, they could be isolated in their own routing domain, ensuring that test traffic cannot get into the production network. In fact, the test network could even use the same IP addresses as the production network without them stepping on each other. A pf(4) ruleset could be written that lets management/administrative traffic from the production network into test. A ruleset could also be written that allows the test systems to talk to a specific management or file server in the production network. If overlapping IP space is used, traffic between the rdomains must be NAT’d as outlined above.

Routing domains can also be used to connect to multiple ISPs. Since userland tools such as dhclient(8) work properly within routing domains, each ISP interface could be put into its own routing domain without the risk of conflicting default routes.

Three OpenBSD Routing Domains

Here if vic1 is connected to ISP#1 and vic2 is connected to ISP#2, the pf(4) ruleset would control which ISP connection to use when users in rdomain 0 connect to the Internet. This provides a much more elegant solution than the outbound load balancing example I wrote about in the PF User’s Guide.

The only shared component of a multiple-dhclient(8) setup is the resolv.conf(5) file. Each copy of dhclient(8) will update the file as it renews its lease.


By virtualizing the OpenBSD routing table you can create virtual routers and/or firewalls within the same physical OpenBSD machine. Networks can be safely isolated from each other without having to worry about traffic crossing network boundaries or IP addresses overlapping. Routing domains can be created by binding one or more interfaces to a routing table so that all traffic crossing those interfaces is automatically forwarded based on the routes present in the virtualized routing table. Traffic can be leaked between routing domains by using the granular pf(4) packet matching syntax to allow policy-based communication between routing domains.

128 thoughts on “Virtualizing the OpenBSD Routing Table”

  1. Thank you for that insight.

    In your opinion, are routing domains suitable to isolate gif(4) tunnels ? To me if I put gif(4) interfaces in another routing domain, the tunnel won’t turn up.

    1. Hey Denis,

      I haven’t actually tested tunnel interfaces in an rdomain. I would assume that if you’re doing something like

      ifconfig gif0 rdomain 5
      ifconfig gif0 tunnel

      that your local interface would also need to be in rdomain 5 and that you’d need a route to in rdomain 5. Is that how you’re doing it?

      Have you seen the tunneldomain ifconfig(8) option? It’ll let you place the inner tunnel traffic into an rdomain.

      I’m curious now if the gif interface is put into an rdomain and a tunneldomain is not configured whether the inner tunnel traffic would be routed in rdomain 0 or in the gif’s rdomain. That’d be interesting to test.

    2. Hello,

      I did some experiment and it is really easy in fact. “tunneldomain” was the directive I was looking for, thank you for pointing !

      # ifconfig gif0 rdomain 1
      # ifconfig gif0 tunnel tunneldomain 0

      Rtable 0 is the “regular” routing table and rtable 1 is my isolated network reachable over gif0 :)

  2. I used gif and rdomains

    ifconfig gif0 rdomain 5
    ifconfig gif0 tunnel

    In your example, and are in rdomain 0 by default. This was wonderful to use serveral rdomains with ipsec+gif before reyk explained how to use serveral enc(4) interfaces.
    The tunneldomain option is used to place the 2 IP addresses above in the rdomain you want.

  3. Thanks! Great howto and that’s exactly what i was looking for.

    One small questions, How do I make this configuration persistent? I mean how can I make it survive reboot?

    1. Hey Omer,

      Except for the pf policy, everything should be doable in the hostname.if(5) files. Going back to the example network in the article, I would create /etc/hostname.vic1 like this:

      descr "Connection to 192.x network in rodmain 1"
      rdomain 1

      If you needed to add a route to rdomain 0 for a network inside rdomain 1, you could add your route statement to the file as well:

      !route -T 0 add 192.168.1/24 -iface

      If you do it this way just make sure the interface that is assigned to is configured first during boot.

  4. This is a great resource for using rdomains and really helps over using just the man pages. I hope that sometime this will get picked up in the FAQ. I have two questions though:

    1) You mention needing to put a route into the domain for return traffic even if using NAT, but I haven’t run into that using 4.9. Was there a change? My test setup used rdomain 0 for the internal IF and rdomain 1 for the outside. The /etc/mygate is set to the ip of the internal IF for rdomain 0 and a default route pointing to the ISP router is added to rdomain 1 via /etc/hostname !directive. Does this work on accident, or should I add a lo2 and a route to the internal (rdomain 0) net in rdomain 1?

    2) I have also noticed that any time I create an alias in any rdomain the entry is added to rdomain 0 route table and not the one I specify even with a complete line “family alias address netmask broadcast rdomain n” in my hostname file. Is that normal, or am I missing something?

    Sorry for the kind of basic questions, but your post is the first I have run into that does a good job explaining. I am using rdomains to handle a dual-ISP setup, but it is a constant learning process for me.

    1. Hi Russell,

      Thanks a lot for your comments and questions.

      A1. Your setup is actually working exactly as I described it should. The default route you have in rdomain 1 is enough to match the destination of the return traffic. So for a packet on the return path, it’ll be run through NAT so the packet has the internal, rdomain 0 IP as its destination address. That destination is looked up in rtable 1 and matches the default route there. Since the lookup was successful the process continues and pf moves the packet into rdomain 0 where it is sent to the end host.

      A2. I didn’t actually test aliases. That’s a good one. To me, that looks like a bug. Possibly ifconfig is not passing the rdomain to the kernel or the kernel is not adding the host route in the proper rdomain.

      1. Just did some quick testing with alias. I can’t reproduce what you’ve described. As long as my interface is already in an rdomain, all aliases are added correctly and routes are put into the proper rtable. This is on 4.9 and 5.0.

  5. I’m getting “tcp rst” in dual-ISP rdomains mode.
    can you please post detailed working config for dual-ISP with rdomains ?

    1. After talking out of band with Ilya, this turned out to be a case where sshd was running on the OpenBSD router where rdomains were configured and when trying to connect from a non-default rdomain to that sshd instance, the router was responding with a TCP RST.

      One thing the article above doesn’t talk about is sockets. Sockets actually belong to an rdomain. If sshd is started like normal (ie, “/usr/sbin/sshd”), then it opens a listening socket in rdomain 0. As far as rdomain 1 is concerned, port 22 is not listening and the kernel will respond with a TCP RST to any inbound connection attempts to port 22. The solution is to launch an sshd instance for rdomain 1 (ie, “route -T1 exec /usr/sbin/sshd”). This sshd instance will open a listening socket, and because it’s been exec’d in rdomain 1, that socket will be associated with that rdomain.

  6. Awesome, using routing domains enabled me to build a working proxy-ARP setup for my OpenBSD 5.0 router (ISP gw with x.y.z.1/24 directly on-link, no route from ISP to our own box, and ISP gw itself proxy-arp’s the entire internet to itself).

    Separate routing domain for em0 (x.y.z.2/24) to ISP, populate it with `arp -s x.y.z.4-254 pub permanent`, and then set up pf `match in on … to … rtable …` rules on em0 and em1 (x.y.z.254/24 + pub arps for x.y.z.1-2).

    It works perfectly for forwarded traffic – but fails for all locally originated traffic on the default routing domain :(

    In this setup, rdomain 2 (em0 / ISP link) has the real default route (x.y.z.1), whereas rdomain 0 (em1 / LAN) has a fake default route (also for x.y.z.1, which is either alias’d or arp-proxied back to em1). Seemingly, locally generated traffic never ingresses on any interface, and `rtable …` rules only work on ingress :(

    How should the default routing domain and its default route be set up to best keep outgoing traffic from the host itself working, in addition to forwarded traffic?

    1. Hi Tero, thanks for explaining your unique setup. You are right, traffic on the router will not pass through the inbound pf rules and won’t be moved into a different rdomain. The software/daemons on the box need to specify which rdomain their socket should use when they create it otherwise they are stuck in rdomain 0. Lots of OpenBSD daemons have command line or conf file options to set their rdomain. For everything else you’ll have to use “route -Tx exec”.

      For simplicity and to avoid the issue you’re having, it’s probably best practice for everyone to keep the main ISP or the one that doesn’t do anything oddball in rdomain 0. It should be thought of as the default ISP and traffic is only leaked to other ISPs/rdomains as needed. This keeps the rule set simple and allows traffic off the router without issue.

      1. Moving the WAN to rdomain0 doesn’t help, because then the routes/hosts on your LAN/rdomainX become unreachable, in the same way… And besides, in this case, the WAN rdomain is full of /32 proxy-arp’d routes for all the LAN hosts, and those are all “fake”..

        Well, this is starting to feel a little scary even for me, but I actually managed to hack this into a working state by applying a little NAT:

        pass out on $lan_if from self to route lan-egress rtable $wan_domain nat-to $wan_if

        $lan_if being the rdomain0/default route destination, and the default route having a ‘lan-egress’ -label attached – very neat way to do the basic `match out on $lan_if to route rtable $wan_domain` btw.

        Remaining problem I still have is that TCP RST’s generated by `block out on $lan_if …` (i.e. after a `match in on $wan_if … rtable 0` change) are sent out on $lan_if… and the sender behind $wan_if ends up getting an `icmp host … unreachable` reply for blocked ports :/

        1. Nice trick with the route label.

          Curious what kind of traffic the router is initiating that is causing you this problem?

          1. 1) TCP SYN comes in on em0, rdomain 2
            2) `match in on em0 … rtable 0`
            3) TCP SYN packet routes out on em1 in rdomain 0)
            4) `block out on em1 …`
            5) pf generates a TCP RST… and sends it out on the rdomain 0 fake default route intended for leaking traffic to rdomain 2 (i.e. out on lo1) – which doesn’t go anywhere, since it never matches the `match in on em1 … rtable 2` -rule used for routing forwarded traffic.

            All in all, it seems that routing traffic between domains doesn’t work all that well :/

            Is there any way to work around this? The generated TCP RST in rdomain 0 doesn’t seem to go through pf at all.

            P.S: this comment box is getting ridculously small

            1. pf doesn’t keep track of which rdomain a packet came from when it moves a packet into a new rdomain. This is why pf isn’t able to automatically route the RST back using the original rdomain. To avoid this you must filter your traffic on the input interface, before the incoming packet is marked with the new rdomain. And this would go for anybody using rdomain leaking, not just you.

              And you’re right, the comment box was getting awfully tight. I changed some CSS around so things look much better now. Thanks.

    1. Hi jofcho,

      All of the components you need to make a dual ISP setup work are talked about in the article. What does your config look like so far? What works and what doesn’t?

      1. Well, I make all settings, and I’m able to transmit and receive data between domains. But how to make something like:
        “pass in on $int_if from $lan_net \
        route-to { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } \

        with rtables?

        pass in on vic1 to rtable 0
        pass out on vic0 nat-to vic0
        pass in on vic1 to rtable 1
        pass out on vic2 nat-to vic2
        How to combine theese two rules?

        1. I can’t think of a straightforward way of doing that.

          Keep in mind that rdomains aren’t designed to do what you’re doing. They’re meant to provide isolation at Layer 3 (and below). You’re trying to do round robin routing to balance Internet use. Stick with the ‘route-to’ method.

  7. Actually I try to make load balancing of outbound connections and use two ISPs.
    Maybe this citation misleaded me:
    “This provides a much more elegant solution than the outbound load balancing example I wrote about in the PF User’s Guide.”

    1. Ah, I can see why that would be misleading. I meant more in the sense that the pf ruleset is a bit cleaner and that you can do the dual-ISP setup using dynamically assigned Internet IPs with rdomains whereas with route-to, you’re pretty well stuck needing static IPs (because you have to specify the gateway IPs in the ruleset).

  8. Do you perhaps have a example of overlapping subnets for rdomain 1 and 2?

    in my case, I’ve setup vlan10 in rdomain 10 and vlan11 in rdomain 11 both have the same network and same gateway ip of I can ping both using the ping -V10/11 and also the hosts located on the 2 subnets. (vlan10/host = and vlan11/host =

    in pf.conf – I’ve added the following,

    pass in on vlan10 to rtable 0
    pass in on vlan11 to rtable 0

    This allows ping from both and .4 hosts to my external interface em0 (

    But now I would like to access the host from a host by accessing a natted IP of say> (rdomain 10).

    1. Hi Danie,

      That’s a good scenario. I haven’t tested this, but here’s what I’ve got off the top of my head.

      # push tcp/80 traffic into rdomain 10 and do dest addr translation
      pass in on em0 proto tcp to port 80 rtable 10 rdr-to

      # get traffic from rdomain 10 destined to 172-net back into rdomain 0
      pass in on vlan10 to rtable 0

      # setup reverse route in rdomain 10
      ifconfig lo10 rdomain 10
      route -T 10 add

      # (no route needed in rdomain 0)

      Let me know what you think. I’m interested to know if this works or not. Sounds like you might have all this already except for the first pf rule.

      1. Hi Joel,
        thanks for the reply.

        The push traffic to rdomain and local loopback for each rdomain did the trick.

        I’m still getting used to the idea of rdomains and pf.


  9. A few additional notes from testing this in 5.1–

    1) You cannot add a route to a table until an interface has been brought into the table.


    $ sudo route -T 20 add
    route: routing table 20: No such file or directory
    $ sudo ifconfig vlan200 create rdomain 20
    $ sudo route -T 20 add
    route: writing to routing socket: Network is unreachable
    add net gateway Network is unreachable
    (well, its unreachable because I didn’t bother to add anything pointing that way, but you get the idea).

    2) For putting apps into an rtable, you don’t have to specify it on a socket level. You can use setrtable(). Check the man page for details, but this is what “route -T x exec” uses.

    if (setrtable(10)) {
    printf(“Cannot set rtable”);

  10. Hey Joel,
    I have a scenario with 2 routers (ospf+ipsec) and two different rdomains (there will be more soon).
    However, just to make sure, is it possible to redistribute routes from all rdomains to the neighbor from a single ospfd instance?
    I did not find any similar scenario to this so far.

      1. Rod, would you share you solution?

        If I was looking at this, I would probably run multiple ospfd instances each with its own ospfd.conf and use the “rdomain” config option to specify the table for each instance.

        1. Sure thing,
          What I did was to set up the rdomains on one router only (eg. Router-1) and created correponding ospfd.conf files for such rdomains (eg ospfd0.conf).
          On the edge router (Router-2) I have only the rdomain 0 propagating only the default route to the other one.

          However I am stuck at a weird issue right now.
          I’ve set up these multiple GRE interfaces on both (the ones for OSPF instances), and I have this weird behavior:

          * When trying to ping the other end of the tunnel:
          1 – The packet leaves the Router-1 from GRE interface (eg. gre0);
          2 – The same packet arrives on Router-2 over another tunnel interface, which makes the traffic being blocked by uRPF or any other antispoofing rules in place.

          * IPSec tunnel in in transport mode, only one GRE tunnel over this is ok, but more than one I am facing the issue stated above.

          Any advice will be appreciated!


          1. Hey Rod,

            From your description I’m not really sure what’s going on. I’d have to see the routing tables, interface configs, a topology diagram, etc, to get an idea of what might be happening.

  11. Hi Joel,
    The diagram is the following:

    (R1) —- [IPSec tunnel] — (R2)
    | |

    R1 has the rdomain feature available the other end (R2) not.

    The IPSec tunnel is over a /30 link between the two (rdomain 0)
    gre1 eg rdomain 1
    gre2 eg rdomain 2 and so on…

    When I have only one gre tunnel, it works perfect, when I put up the other one, only the last one which is enabled works.
    What happens: pinging from R1 (ping -V1 x.x.x.x w.w.w.w), the packet leaves the correct gre interface but it arrives in the other end at gre2 for an example (both have uRPF enabled causing the packets to be blocked).

    BTW, R2 pf states are all if-bound byt setting in the options.

    Any hints?

    1. Are you using different outer IP addresses on the tunnel interfaces of R1? Other than that nothing obvious pops out but there’s still details I can’t see from what you’ve posted. I would try posting to the misc@ list and hope that someone else has done something similar.

      1. Well basically, it is set up like this:

        gre0 tunnel
        gre0 inet netmask
        rdomain 1

        gre1 tunnel
        gre1 inet netmask
        rdomain 2

        They are directly connected by a /30 link ( (R1) (R2))
        IPsec is upon these interfaces and all SAs seems to be ok

        Did a test without IPsec and got the same results.

        PS: the R1 I only have rdomain 0. I only want to distribute de default route via OSPF.

        1. So from this, it looks like R2 is seeing two tunnels coming from Is that right? That could be it right there. R2 might be having trouble multiplexing the GRE traffic because the source address of each tunnel is the same.

          1. Yeah, that could be it.
            Maybe by using alias could fix this? I will give it a shot, also tried other options like disabling pf, using the default rdomain 0 for all the interfaces etc.

            I will try this, but I will also need to set up another IPsec tunnel using also the interface alias?

            Thanks for your help Joel!


          2. Well, got this fixed. not the way I like tho.

            But what I did, was to turn off uRPF for both tunnel interfaces in both devices.

            Also created an alias for the parent interface and established another SA with te alias as well.

      2. Just to be clear I am correcting the diagram as follows:
        (R1) —- [IPSec tunnel] — (R2)
        | |

  12. Hi Joel,

    This is really great resource and really helps over understanding the rdomains concept.Thank you very much .

    Unfortunetly, i couldn’t solve my problem from these examples.

    This is as follows ;

    I have 3 rdomain (r0,r1,r2) . r1 and r2 must be seperate each other (they have to communicate between aech other from their default gateways.

    I’m using haproxy in r0 . HA must listen both r0, r1 and r2 rdomain interfaces .

    Problem is I have no chance to start multi HA instance for all rdomains.

    So, is there any way to listen all interface IPs in r0 domain ? Thank you for your kind advice & support.



    1. Hi Omer,

      Thanks a lot for the feedback!

      I don’t really understand what you’re asking about haproxy. Can you try explaining it again? You say that you’re using haxproxy in r0 and it must listen in r0, r1, and r2 and you’re wondering how to make it listen on all r0 interfaces? *confused*


      1. Hi Joel,

        Thank you for your kind support&consideration.
        Please find my network diagram at ” ” .

        My problem is ;
        trafic from server0 to server1/server2 must be go from routers (not inside of my bsdbox.) So I have to use rdomains.

        But when I use the rdomains, my haproxy daemon can work just in one rdomain and I do not want to open another haproxy instance in other rdomains because of the session and cookies sharing problems.

        Is there any way to give a permission to a daemon for to listen/talk all rdomains interfaces ? or is there any similiar solution for this kind of problem ?

        P.S: Sorry for my poor English :(

        Thank you .


        1. Hi Omer,

          Your english is just fine :) Thank you for the diagram, that makes things much clearer. I’m not sure if it’s possible to do what you need. If haproxy doesn’t support being virtualized, then your only choice is to run an instance in each rdomain. I also looked at relayd(8) in OpenBSD but it doesn’t seem to support virtualization either. (On a side note, have you evaluated relayd?)

          What is the session and cookie sharing problem that you mentioned?

          1. Thank you for your kind support Joel :)

            Sorry I never use relayd before . But when I examine relayd , I saw that backend balance algorithms types are less than Haproxy. Bytheway, I couldn’t find any document for “relayd performance vs haproxy performance”

            Best Regards,


  13. Hi Joel!
    You’re doing great job, many thanks to you :)

    Do you know if there are any unexpected surprises with vlan interfaces and different rtables? I am planning to make a lab in some time, but for now I have no idea how it works when you separate some vlans from others at one physical interface.

    1. Hi Roman,

      Thanks for the feedback :)

      Putting vlan interfaces into different rtables should be no different than doing it with physical interfaces. You should find that things behave exactly as I described in this post. The rdomains on vlan interfaces will be in effect even if the vlans are riding on the same physical interface. Remember, rdomains provide separation at Layer 3. Anything below that is irrelevant. Good luck with your lab!

  14. Yeah, exactly as you said. Going to test it on a real hardware now.
    Do you mind if I make russian version of this article (for OpenBSD 5.4), including references and stuff :)?

    1. You’re free to translate the article, keeping the copyright notice intact and a clear link back to the original URL.

  15. “Virtualizing” /etc/resolv.conf is not supported, is it? How could I set different resolvers for each rdomain? The nameserver is not reachable from all rdomains so processes in those rdomains can’t resolve hostnames.

    How to deal with that?


    1. That’s a really good question.

      You’re right, resolv.conf cannot be virtualized. You’re stuck with one set of resolvers for all rdomains.

      As a solution, what about using pf.conf to push DNS traffic into the rdomain where the servers are reachable? You could combine this with NAT and/or RDR if your rdomains have overlapping IPs.

  16. I am attempting to use rdomains to provide isolation for multiple redundant subnets for example rdomain 10 rdomain 11 That part I understand but then I want to do a one to one nat or binat with internface em0 rdomain 0 Rdomain 10 and 11 will have multiple servers each needing their own public ip..
    server 1 rdomain 10
    server 2 rdomain 10
    server3 rdomain 11
    server 4 rdomain 11

    rdomain 10 and 11 don’t need to be able to communicate with each other, but its ok if they do so on the rdomain 0. Here is what I have so far.

    rdomain 3020
    vlandev em1
    !route -T 3020 add
    !route -T 0 add -iface

    #client to private
    match in on em0 to rdr-to rtable 3020
    match out on vlan3020 nat-to vlan3020

    #server to egress
    pass in on vlan3020 to rtable 0

    *match out on em0 received-on vlan3020 from nat-to

    This part doesn’t work, but I don’t know how to fix it. I hope you understand what I am trying to do, sorry if I should based on the information above, but I must be missing something.

    1. Hey James.

      I don’t understand the purpose of this line:

      match out on em0 received-on vlan3020 from nat-to

      em0 is on the network right? So this rule is trying to match traffic coming from in rdomain 10 and going out em0 (in rdomain 0). I don’t understand the “from” part. Where is getting NATed to And then where does come into play? It looks to me like this rule would never match because the source IP would never match

      1. Your right that statement is wrong let me try to explain. I am thankful for your response!

        I need a true one to one nating to work both directions. I am going to have multiple rdomains which will have the same subnets ie the I will have one public interface em0 for example My logic problem is that in each of the rdomains 3020, 3021, 3025 … etc. I will have a address which needs to be mapped to its address. The same IP that you use to connect to the system from rdomain 0 needs to be the same IP that system uses to access systems on rdomain 0 side of the network. I don’t want to round robin or to just nat to the public interface of the em0:0. In order for this to work, I need to be able to say pass this traffic out to em0. Second I have to know the private IP that was used and which rdomain it came from so that you can nat to the correct public IP on em0. To further clarify there would also be a rdomain 3021 and there would be another private IP of I would need to be able to distinguish between the two rdomains and private IPs so that I would know which public IP on em0 both should be nated to. To further complicate the situation there will also be multiple systems in each rdomain ie 3020 may have up to 10 systems so you can’t just say if you can from rdomain 3020 nat to this public IP on em0. You have to know which private IP it is even within the rdomain so that you can map it to its corresponding IP on em0.

        so something like.
        #server to egress –need help here
        pass in on vlan3020 to rtable 0
        match out on em0 received-on vlan3020 from nat-to
        match out on em0 received-on vlan3020 from nat-to

        pass in on vlan3021 to rtable 0
        match out on em0 received-on vlan3021 from nat-to
        match out on em0 received-on vlan3021 from nat-to

        #em0 to private this part makes sense and will work
        match in on em0 to rdr-to rtable 3020
        match in on em0 to rdr-to rtable 3020
        match out on vlan3020 nat-to vlan3020

        match in on em0 to rdr-to rtable 3021
        match in on em0 to rdr-to rtable 3021
        match out on vlan3021 nat-to vlan3021

        1. Hey, no worries. You’ve got a really interesting setup. I’d love to know what you’re running there that requires this configuration.

          Now I’m not sure about this rule:

          match out on vlan3020 nat-to vlan3020

          Initially you’re passing traffic from the servers to 10.2.0.x but on the way back you’re NATing the source IP to something in 192.168.64.x. It seems like that will cause breakage.

          What exactly is not working at this point? Your explanation of the topology is good. I’m just not sure what isn’t working.

          1. well it has be nated both directions. The problem with the above statements is that ‘match out on em0 received-on vlan3021 from nat-to’ is not a valid statement and I can’t figure out the correct syntax to make that work. I was hoping that you might have an idea. It seems like it should be supported with all the other functionality they have built around rdomains. I have looked at the pf.conf man page and tried but haven’t figured it out yet hopefully It can be done and I will feel silly for struggling with it.

            This is for a test environment where we don’t want to change the IPs of the systems. We also want to be able to monitor the traffic easily from those system across the network and be able to tell which system it is. If I nat to a single IP or round robin on em0 I lose a level of visibility.

            1. so, I finally got the nat part to work, but now its still not working I am using pass rules. The symptom that I am seeing is that when I start the connection from the nat-ed side then it get’s nated and goes out and starts the connection to the external system, but on the response part of the request its using the same rule it went out on even though the rule is only an out only rule. I don’t really understand that. I have tried match rules, but haven’t gotten that to work either. If its a new connection it uses the correct rule and I am able to access the nat-ed system from an external system. I would like to understand this or is that a bug and if so do you know how to report it?

              1. Hey James,

                Without seeing any sort of outputs or troubleshooting steps, I could only guess what’s going on.

                There’s something to be said for operational simplicity :-) You may want to take a step back and see if there is a more straightforward way to design this system.

  17. Hi Joel,

    Great article! Thanks for putting this together. For me personally you are slowly becoming the go-to source for OpenBSD related questions. Keep up the good work.

    I am trying to use the rtable functionality in a 2 ISP setup in which part of the network uses one ISP and the rest is using the second one. I am running 5.3 in a HA setup with carp interfaces. I got stuck pretty early in my tests. I was able to create the additional rdomain 1 route table but when I try to add interfaces to it I get:
    ifconfig em0 rdomain 1
    ifconfig: SIOCSIFRDOMAIN: Invalid argument.
    Have you seen this type of error message before? Do you think is related to the fact that the interface I am trying to add to rdomain 1 has a carp interface associated with it? What would be the steps for migrating a non multi-routing table HA setup to a multiple rdomain HA setup?

    Thanks for your help with this.

    1. Thanks a lot for the feedback, Adrian. Appreciate that.

      Can you post the output of “ifconfig” and “route -T 1 show” please?

  18. Here you go:

    ifconfig em0
    em0: flags=8b43 mtu 1500
    lladdr 00:50:56:a3:59:cb
    priority: 0
    media: Ethernet autoselect (1000baseT full-duplex,master)
    status: active
    inet6 fe80::250:56ff:fea3:59cb%em0 prefixlen 64 scopeid 0x1
    inet netmask 0xffffff00 broadcast

    route -T 1 show
    Routing tables

    Destination Gateway Flags Refs Use Mtu Prio Iface
    default UGS 0 0 – 8 em0

    Gave it a try with removing the carp interface and trying to associate the interface with rdomain 1 again but still no go. So I do not think the error message is related to the presence of the carp interface.

    Thanks for looking into this.

    1. Should have also mentioned that I am running the two servers in VMware ESXi 5.1. I’m starting to believe it might be a VMware NIC driver limitation.

      1. Virtual or physical shouldn’t matter since the rdomain goo is entirely inside the network stack and doesn’t touch the physical drivers at all. I can add my em(4) interface to an rdomain without issues on a VM.

        How did that default route get into rtable 1?

        1. route -T 1 add

          Made some progress after destroying the carp interface and rebooting. I have now the em0 interface in rdomain 1. I also added one vlan interface to rdomain 1. I will keep at it and will give you an update once this is done, but at this point it looks like the reboot helped.

  19. Hi Joel
    I am trying to essentially combine a bunch of NAT routers into one.
    I have traffic being routed to me on em0 (
    And I have vlans on the inside em1 using the same IP range (

    vlan42: flags=28843 rdomain 1 mtu 1500
    lladdr 00:50:56:96:65:47
    priority: 0
    vlan: 42 parent interface: em1
    groups: vlan
    inet netmask 0xfffffc00 broadcast

    vlan43: flags=28843 rdomain 1 mtu 1500
    lladdr 00:50:56:96:65:47
    priority: 0
    vlan: 43 parent interface: em1
    groups: vlan
    inet netmask 0xfffffc00 broadcast

    Then I do this in pf.conf

    ## rdomain 1
    pass in on egress inet from any to rdr-to rtable 1
    pass in on vlan42 inet from to ! nat-to rtable 0

    ## rdomain 2
    pass in on egress inet from any to rdr-to rtable 1
    pass in on vlan42 inet from to ! nat-to rtable 0

    It all worked great I thought. I could use RDP and SSH from the “outside” to thos IPs and they connected to the correct machine.

    But I am not able to connect from the “inside” to the “outside”.
    Ie. I can not ping to the DNS server on the outside from node on the inside.

    1. Hi Kjell,

      Just a question first: is this correct that the vlan42 and vlan43 interfaces are in the same rdomain and also have the same IP address?

        1. Ok, no problem.

          – Does this happen on both rdomain 1 and 2?
          – Can you double check the pf.conf rules you pasted above; the one comment says “rdomain 2” but the rules are for rtable 1. It also refers to vlan42. Just make sure it’s an accurate copy/paste.
          – Have you checked your whole pf.conf to see if there’s a rule that is blocking connections initiated from inside->outside?

          1. Hi yeah I see that there are some copy paste errors.
            I have made new copies on slexy that I know are right.
            Since the same problem appears on both 42 and 43 I have included only the stuff related to 42.

            # ifconfig ->
            # vim /etc/pf.conf ->
            # route show – >
            # route -T 1 show ->

            As I said, I can RDP or SSH into the machine with from the outside and it can ping hosts on the outside, but can not http, ftp or anything else.
            There are no other rules or firewalls to block this, as the router itself has no problem fetching a file with FTP.

            1. And I fixed it.

              pass in on egress inet from any to rdr-to rtable 1
              pass in on vlan42 inet from to ! nat-to rtable 0
              This ruleset only work one way.

              pass in on egress inet from any to rdr-to rtable 1
              pass on vlan42 inet from to ! binat-to rtable 0
              This works.

              I changed the nat to a binat rule and now everything works :)

  20. Is it possible to send traffic out different interfaces based on src IP in FreeBSD 10? I’ve been trying to accomplish this, but can’t seem to find the right commands using pf. Any tips would be much appreciated.

    1. Hey Mike. I’m not familiar with all of FreeBSD’s bells and whistles in their network stack, but with pf you’ve got options like route-to and reply-to which can policy route traffic. There might be equivalents in ipfw too.

  21. Very good article. I still have one question though. Is rdomain and rtable one-to-one mapping? I just don’t see a command in your post to link rtable with rdomain directly.

    1. Hi Yang,

      To be honest, I don’t understand the reason behind the terminology. The way I think about is:
      – An rtable is a specific routing/ARP/ND table instance
      – An rdomain is the collection of an rtable and the network interfaces that are bound together

      The rdomain is the larger construct inside of which is an rtable. So far that works for me and helps me keep it straight.

  22. Good morning, I work at Intituto Tecnologico de Tuxtla Gutierrez, Chiapas, Mexico, is an school & I have 4 links for go out, I work at Ing. en sistemas computacionales (one o 8 careers), I have an OpenBSD box with 6 network interfaces, one o these is with vlans, I want to declare 4 rdomains,
    rdomain1 : bge0 –>nat–>rl0
    rdomain2 : vlan13 –>nat–>em3
    rdomain3 : vlan14 –>nat–>em1
    rdomain4 : vlan15 –>nat–>em0

    In addition I wanna that vlan14 has a proxy filter (squid) and We need in bg0 lan access some sites in em0 lan

    up descr VLAN_TRUNK group VLAN_TRUNK
    inet NONE descr redmmto group mmto vlan 12 vlandev em2
    inet NONE descr red13 group treco vlan 13 vlandev em2
    inet NONE descr red14 group cator vlan 14 vlandev em2
    inet NONE descr red15 group xxr vlan 15 vlandev em2

    Please can you help to me, can you give me some examples, directions, help!!!!!!!!!!!!!!

    1. Hi Jorge, I think for what you’re trying to do it would take more than a comment on a blog to get you going. You most likely need dedicated help from someone that can work through all the details of what you want to accomplish. Maybe you could try meeting at a local BSD user group and finding someone to help. Or maybe post an ad online for an OpenBSD consultant.

  23. Hi to everybody.
    First of all sorry for my English…
    I have a problem with my openbsd 5.8 firewall…

    I have 3 NICs (em0,em1,em2) for my WAN connection and 2 NICs (em3,em4) for LAN subnet and SERVER subnet.
    My ISP give me 16 public IPs address:
    – subnet: a.b.c.64/28
    – gateway: a.b.c.65

    I want use configure my WAN interface as:
    – em0: a.b.c.70/28 for LAN internet navigation
    – em1: a.b.c.71/28 for SERVER internet navigation and for contact server from outside
    – em2:a.b.c.72/28 for VPN

    The internal interface configured as:
    – em3: for LAN subnet
    – em4: for SERVER subnet

    LAN subnet should bet natted by a.b.c.70 while SERVER subnet should be natted by a.b.c.71
    I would like also that some internal server (email) can be contacted by external users using the ip a.b.c.71

    The problem is that LAN and SERVER are natted by the same IP (a.b.c.70) and I can connect server only using the first public IP address (a.b.c.70) and NOT using the second (a.b.c.71)

    My /etc/hostname.em0 is:
    inet a.b.c.70
    !route add default a.b.c.65

    My /etc/hostname.em1 is:
    inet a.b.c.71
    !route add default a.b.c.65

    My /etc/hostname.em2 is:
    inet a.b.c.72
    !route add default a.b.c.65

    My /etc/hostname.em3 is:

    My /etc/hostname.em4 is:

    In my pf.conf I have set 2 NAT rules:
    EXT_if0 = “em0”
    EXT_addr0 = “a.b.c.70”

    EXT_if1 = “em1”
    EXT_addr1 = “a.b.c.71”

    INT_if3 = “em3”
    INT_addr3 = “”
    INT_subnet3 = “”

    INT_if4 = “em4”
    INT_addr4 = “”
    INT_subnet4 = “”
    SERVER_addr = “”

    match out log on $EXT_if0 inet from $INT_subnet3 to any nat-to $EXT_addr0

    match out log on $EXT_if1 inet from $INT_subnet4 to any nat-to $EXT_addr1

    pass in on $EXT_if1 inet proto tcp from any to $EXT_addr1 port 25 rdr-to $SERVER_addr

    #Permit LAN internet navigation
    pass in quick on $INT_if3 inet proto { tcp, udp, icmp } from $INT_subnet3 to any

    #Permit SERVER internet navigation
    pass in quick on $INT_if4 inet proto { tcp, udp, icmp } from $INT_subne4 to any

    How can I solve my problems? Rdomain can help me?

    Thank you…

    1. Hi Michele,

      I think you could solve your issue my using rdomains, however I don’t think it’s necessary. You could simplify your config and get it working much easier than overlaying rdomains.

      The issue you’re running into is an order of operations issue. The way the IP stack works is that ROUTING is first, NAT is after. Imagine a packet coming in on em4 from a server and going to the Internet. The first step is for the kernel to make a routing decision. You’ve got 3 default routes, but it appears the route going out em0 is being chosen. So the kernel puts the packet in the egress queue for em0 and runs the packet through pf again. _Now_ your NAT rule matches, since the packet is getting ready to send on em0. And since your NAT rule for em0 says to nat to .70, that’s what happens.

      That’s the logic of what’s happening. To fix it, you could do a few different things but the easiest way in my opinion would be to simplfy how you’ve arranged your Internet interfaces.

      Instead of having 1 IP per inteface, designate one interface for Internet and configure it with .70. Then add .71 and .72 as aliases on the same interface. This eliminates the routing complexity since you’ll now have exactly 1 egress interface to the Internet, all traffic (from servers and LAN) will hit that interface and you can NAT the traffic very easily with your nat-to rules.

  24. Hi Joel,
    I was wondering if you could help me out. I have an openbsd gateway at home and I’m trying to use rdomains to make all traffic going to the internet or coming from the internet route out of the openbsd gateway through a freebsd server for some packet inspection (bro) before getting routed back to the gateway and onto the final destination.

    I was able to get it working by having the internet facing pppoe connection and a vlan interface pointing to the freebsd server in rdomain 1 and the freebsd server with a second vlan pointing to
    rdomain 0 on the obsd gateway.

    My problem is the openbsd gateway itself is exhibiting some odd behaviour. The gateway is serving dns via unbound and other clients on the network can get dns and access the internet etc. but the openbsd gateway itself can’t seem to resolve dns for itself. When logged into the gateway I can’t seem to ping my own interfaces by IP address (besides the loopback) however other hosts on the network can ping them.
    internet <- pppoe0,rdomain1 -(obsd-gateway)-vlanX,rdomain1 freebsd vlanY,rdomain0 – (obsd-gateway) – vlan(a|b|c|etc)rdomain0
    There are static routes and default routes in the corresponding rtables.

    I could ping vlanX,rdomain1 interface from rdomain0 through freebsd, but that’s about it.
    I tried going back to a flat single (rdomain0) routing table and realized I still couldn’t ping my own interfaces.
    I thought I would reach out and see if you had any idea what could be going on regarding the pinging my own interfaces issue and the obsd gateway when using rdomains being unable to resolve dns for itself.

    1. Hey Aaronn,

      That’s an awesome use of rodmains. Nicely done.

      I’m a little confused on what the precise problem is. Is it DNS from the OpenBSD gateway? Or that the gateway cannot ping itself? Can you be a little more precise?

      For DNS, what are the contents of /etc/resolv.conf? If using a local resolver, what rdomain is it listening in? What rdomain are your client applications in that are trying to do DNS resolution?

      For pinging, are you pinging within the rdomain? Do you have pf rules that might be blocking this traffic?

  25. Hi Joel, thanks for getting back to me. I managed to get it working by fixing my pf.conf file, I think my pf.conf file is getting too complicated, a firewall rule overhaul is due. My gateway can resolve and ping out now though. However the second issue is still a problem, and it was a problem before I started using rdomains about the gateway pinging it’s own interfaces within the default rdomain. Anyways that’s not a big deal just weird.

    However now that I’ve successfully migrated to this configuration my ipsec vpn seems to be broken. Before moving to rdomains it worked fine with a config like this:

    ike passive esp transport \
    from to any \
    main auth “hmac-sha1” enc “aes” group modp1024 \
    quick auth “hmac-sha1” enc “aes” \
    psk “” \
    tag IPSEC_IKE1_IN

    pass in quick log on $if_extern inet proto udp from any to $gate_static1 port $svc_ipsec keep state
    pass in quick log on enc0 inet proto udp from any to $gate_static1 port 1701 keep state (if-bound) tagged IPSEC_IKE1_IN
    pass quick log proto { esp, ah } from any to any

    where $if_extern is my internet facing interface, $gate_static1= and $svc_ipsec is a macro port group for 500 and 4500

    Anyways this is a L2TP/IPSEC configuration (for myself and my wife’s android phones) which was working before I started using rdomains. Now that I’m using rdomains it’s not, from my reading it looks like it’s because my rdomain1 is my internet connection which is the interface ipsec was listening on. It looks like it is possible to have the daemon start in rdomain1 but the npppd daemon doesn’t play well with rdomains yet. I thought I would try and get around this by port forwarding from rdomain1 to an internal gateway IP on rdomain0 (going through the freebsd server). Anyways, I can’t seem to get this working. I suspect I’m not configuring it right. Before the ipsec tunnel was terminating on the edge gateway, now with rdomains it basically needs to terminate on the gateway on the internal rdomain (rdomain 0). I suspect this should be possible Do you know what changes to my config are needed to make it work? Thanks again,

    1. Hey Aaron,

      Are you sure this line is correct?

      pass in quick log on enc0 inet proto udp from any to $gate_static1 port 1701 keep state (if-bound) tagged IPSEC_IKE1_IN

      I thought enc(4) always presented ipencap packets. That’s how I’ve always created such rules.

      Do you have any idea where it’s failing? There’s a lot of components involved in getting one of your tunnels up. Is isakmpd responding? Does it get as far as even trying to communicate with npppd?

      1. Hi Joel,

        I think you only need proto ipencap for tunnel mode, I’ve been doing transport mode. It worked before with udp as the proto, anyways it doesn’t look like the problem is there anyways if you see the debug log further below: here is my ipsec.conf config
        ike passive esp transport \
        from publicip (rdom0rfc1918IP/30) to any \
        main auth “hmac-sha2-256” enc “aes” group modp1024 \
        quick auth “hmac-sha2-256” enc “aes” \
        psk “notmyrealpsk” \
        tag IPSEC_IKE1_IN

        and the pf.conf rules:
        pass in quick log on $if_extern inet proto udp from any to $gate_static1 port $svc_ipsec rdr-to $gate_vlan3 keep state
        pass in quick log inet proto udp from any to $gate_static1 port 1701 rdr-to $gate_vlan3 keep state (if-bound) tagged IPSEC_IKE1_IN
        pass quick log proto { esp, ah } from any to any

        If I look at the logs with a bit of debugging on I can see the flows are getting established, same if I do ipsecctl -s all
        however if I run tcpdump -i enc0, or run the npppd daemon logging to stderr I don’t see anything, I don’t think the traffic is getting to the npppd daemon. Any ideas?
        any ideas?

        1. Ahh right you are with transport vs tunneled. Thanks for the clue bat :)

          You said flows are getting established. Can you show me the data/outputs that you’re looking at?

  26. Looking over my post I see some portions of the configs I posted were filtered out so my ipsec.conf config looks broken, probably because of the angle brackets I was using. I had substituted the values for my static IP and psk, so the lines that refer to them do have values there, like in ipsec.conf:
    from mystaticip to any \

  27. Hi Joel,
    In case you are still around to answer questions on this, do you happen to know any IPv6? I tried replicating the first part using IPv6, but my packets from rdomain 0 back to rdomain xx, seems to get dumped. Adding a route to localhost will just loop the packets there until the TTL reaches 0. I made a post on openbsd-misc describing the set-up, but without any luck so far:

    Do you have any ideas? :)

    1. Hey Claus,

      Interesting problem. I’m not totally surprised you’re seeing issues with IPv6 since the v6 stack gets far less exercise than the v4 stack. Unfortunately, the v6 stack does sometimes have different behavior or even latent bugs.

      Out of curiosity, what version of OpenBSD is the router running?

      What does ‘pfctl -vvss’ show for the icmp6 traffic at the time of the test?

      Can you fiddle around with this route “route -T 0 add 2a01:7e8:35:fab::/64 ::1” and instead of pointing it to the loopback, point it to an interface, make it directly connected with -iface, and whatever else you can think of? The “time exceeded” message makes it look like the return traffic is looping inside the gateway.

    2. I replicated the same issue myself. Given that my use case is policy based routing, I switched to the pf’s “route-to” directive, which works okay for IPv6.

      Anyway, I tampered with some options like pf’s “no state”, and something behaved differently with that option, and some messages showed up in dmesg (sorry I forgot to save the details, but that’s not informative either). None of these options really solved the problem with rtable, however.

  28. Hi Joel,

    First of all, thanks for the quick response! :)

    I’m running 6.1 on the router.

    To start with the time exceeded message, if I tcpdump on lo0 I can see the message looping in there, so I’m certain that message is looped on the localhost interface.

    pfctl -vvsr gives:
    # pfctl -vvsr
    @0 block return all
    [ Evaluations: 8 Packets: 0 Bytes: 0 States: 0 ]
    [ Inserted: uid 0 pid 96072 State Creations: 0 ]
    @1 pass all flags S/SA
    [ Evaluations: 8 Packets: 10 Bytes: 680 States: 1 ]
    [ Inserted: uid 0 pid 96072 State Creations: 5 ]
    @2 block return in on ! lo0 proto tcp from any to any port 6000:6010
    [ Evaluations: 8 Packets: 0 Bytes: 0 States: 0 ]
    [ Inserted: uid 0 pid 96072 State Creations: 0 ]
    @3 pass in on em2 inet6 from 2a01:7e8:35:fab::/64 to 2a01:7e8:1:800::2fd flags S/SA rtable 0
    [ Evaluations: 5 Packets: 76 Bytes: 7904 States: 1 ]
    [ Inserted: uid 0 pid 96072 State Creations: 1 ]
    @4 pass out on em1 all flags S/SA
    [ Evaluations: 8 Packets: 116 Bytes: 13816 States: 1 ]
    [ Inserted: uid 0 pid 96072 State Creations: 2 ]

    The biggest clue I find here is that I can see the packets leaving, @3 for jumping rtable, @4 for sending the packet on to “the internet”, and @4 again for messaging towards “the internet”. I cannot see anything that would indicate the return traffic (like having @1 triggered a whole lot more).

    I tried adding a route to the rdomain 75 network via the em1 interface address (it’s a /126, so I don’t really have a lot of wriggle room):
    # route -T 0 add 2a01:7e8:35:fab::/64 -iface 2a01:7e8:1:800::2fe

    This does not fix the error, but gives something I haven’t seen with tcpdump before:
    07:59:15.788933 2a01:7e8:35:fab::2 > 2a01:7e8:1:800::2fd: icmp6: echo request [flowlabel 0xe1717]
    07:59:15.789211 2a01:7e8:1:800::2fd > 2a01:7e8:35:fab::2: icmp6: echo reply
    07:59:15.790331 2a01:7e8:1:800::2fe > 2a01:7e8:1:800::2fd: icmp6: 2a01:7e8:35:fab::2 unreachable address

    Just for the fun of it, I tried changing the route to having the network address as destination, but it gave the same result:
    # route -T 0 add 2a01:7e8:35:fab::/64 -iface 2a01:7e8:1:800::2fc

    My best guess is that whatever function that, in v4, moves the packets from rdomain 0 to the right rdomain via a route (pf?), is not triggered in my v6 case.

    Could it be that the behaviour for v6 has changed without getting documented?

    Again, thanks!

    / Claus

    1. Damn, that’s too bad it didn’t improve by changing the route.

      Are you saying this worked prior to 6.1? There was an awful lot of work that happened in the network stack leading up to 6.1 (and the work continues for 6.2). So yeah, maybe this is a regression. I wonder if this still works for v4? I have a box here doing v4 inter-rdomain routing but I had trouble upgrading it to 6.1 so it’s still on 6.0.

      1. Haven’t tried it with a version prior to 6.1, but I did a v4 version and that worked fine (on the same set-up). Actually, I didn’t need the return route on rdomain 0 on v4 for it to work, which surprised me a bit, but if a lot of work has been done, that could be why.

        1. I would follow up with your post on misc with this information: v4 works just fine on 6.1 but v6 doesn’t with the equivalent rule set. Provide a copy ‘netstat -rnf inet6’ and the output of your tcpdump on lo0 that shows the return traffic looping around. Maybe one of the v6 guys will be able to take a look.

          1. So I’ve written a follow up mail on misc, but there haven’t been any reply. For now I have taken rdomains out of the equation, as IPv6 is more important. Thank you for all your help, I hope I’ll get it working in the future!

  29. Hi Joel,
    is there a solution for swapping packets between rdomains on a bridge ?
    I have tried but doesn’t work, different pf rules. Only when the device is acting as a router including nat and so on.
    Setup: 2x phy interfaces and 1x vether on bridge0 in rdoamin 0
    1x phy with ip in rdomain 2, no bridge simple interface. ipforwarding enabled.

    Now I want to map some traffic ( for testing on bridge0/rdomain 0 and send it out on rdomain 2.
    Any ideas ?


    1. Hi Markus,

      Do your pf rules work without modification if you’re running as a router? You’re certain there’s nothing wrong with the rules themselves that’s preventing it from working?

      Have you broken out a sniffer or tcpdump to try work out where the packets are going when running as a bridge? I.e., are they being bridged within the rdomain? Is pf dropping them? Something else?

  30. Hi Joel,
    here is the setup, the config and the results.
    Detail setup of my home lab:
    OBSD6.1, i386
    rdomain0= bridge0 = vr0,vr1,vether0( ,pc- client ( is on vr0. net,gw (paloalto220)
    rdomain2= vr2=, gw = (fritzbox)

    initial test ping from both rdomain controlled with tcpdump
    a) from client:
    # tcpdump -n -i vr1 host and icmp
    tcpdump: listening on vr1, link-type EN10MB
    21:38:46.027991 > icmp: echo request
    21:38:46.047190 > icmp: echo reply
    b) from rd2
    # ping -V 2
    PING ( 56 data bytes
    64 bytes from icmp_seq=0 ttl=59 time=20.458 ms
    # tcpdump -n -i vr2 host and icmp
    tcpdump: listening on vr2, link-type EN10MB
    21:39:53.539804 > icmp: echo request
    21:39:53.558920 > icmp: echo reply

    1. scope outgoing packet (with or without correct nat), return packet later
    simple pf.conf
    changes made in pf.conf are reloaded with
    # pfctl -F all -f /etc/pf.conf
    set skip on {lo1,enc0}
    match out on rdomain 0 to nat-to (vr2) rtable 2
    pass out on vr2 nat-to vr2
    # tcpdump -n -i vr1 host and icmp
    tcpdump: listening on vr1, link-type EN10MB
    21:43:19.877046 > icmp: echo request

    match rule matches and makes nat ,but still in rdomain 0, see capture it is vr1 !
    set skip on {lo1,enc0}
    match out on rdomain 0 to rtable 2
    pass out on vr2 nat-to vr2
    # tcpdump -n -i vr1 host and icmp
    tcpdump: listening on vr1, link-type EN10MB
    21:46:19.188579 > icmp: echo request
    21:46:19.207370 > icmp: echo reply
    0> nothing happens

    set skip on {lo1,enc0}
    pass in on vr0 to rtable 2
    pass out on vr2 nat-to vr2

    still on vr1
    # tcpdump -n -i vr1 host and icmp
    tcpdump: listening on vr1, link-type EN10MB
    21:50:52.458533 > icmp: echo request
    21:50:52.477280 > icmp: echo reply

    now changing the default gw on the client pc from (Palo) to to vether0 ip

    c) still on vr1
    b) with “match” instead pass it is moved to vr2 :-)
    # tcpdump -n -i vr2
    tcpdump: listening on vr2, link-type EN10MB
    21:57:04.374172 > icmp: echo request
    21:57:09.373865 > icmp: echo request

    ok, still not natted, but with match and obsd in route function
    a) now with nat-to in the match rule,see above a)
    # tcpdump -n -i vr2
    tcpdump: listening on vr2, link-type EN10MB
    21:59:22.389258 > icmp: echo request
    21:59:22.408343 > icmp: echo reply

    it works, but still only as router :-(

    In my point, the difference to bridge is that we directly target the obsd box with it ‘s mac adress.
    mybe I/we need other tricks to make route lookups, rdr , route-to or other statements …?


    1. ..based on the pf.conf doc the behavor makes sense:
      Used to select an alternate routing table for the routing lookup. Only effective before the route lookup happened, i.e. when filtering inbound.

      > but on bridge mode we do not make a route lookup, in routing we do.
      adding a rule …route-to rtable 2 isn’t a solution.


      1. … GOT IT !!

        the “quick” and nat-to was missing in the route-to statement.

        with this:
        pass in quick on vr0 to route-to (vr2 nat-to (vr2) rtable 2
        I can run a bridge and forward the traffic into another rdomain.
        sorry for the many posts

        1. :-(
          ok, it works, but still unexpected behavior:
          The first return paket is always dropped, icmp or udp in my case.

          # tcpdump -n -i vr2 host
          tcpdump: listening on vr2, link-type EN10MB
          00:13:03.550705 > 48297+ A? (33)
          00:13:03.575126 > 48297 1/0/0 A (49)
          00:13:03.575264 > icmp: udp port 54179 unreachable
          00:13:03.582128 > 48297+ A? (33)
          00:13:03.600848 > 48297 1/0/0 A (49)

          with ping to the same, the first echo reply is dropped.
          It looks like there is no state created on vr2, only after the 2nd packet.

          set skip on {enc0}
          pass in quick on vr0 to route-to (vr2 nat-to (vr2) rtable 2
          pass out quick on vr2


          1. Are you running pflogd? Can you add “block log” to the rule set and then tcpdump pflog0 just to confirm it’s pf blocking the first reply packet?

            Is there a change in behavior if you had a “pass in quick on vr2 from”?

            1. Hello Joel,
              I have changed the design back to simple “routing” and it works for user traffic entering the device and forward it to the right rdomain.
              However, how can I catch self initiated traffic from the router ?
              I do not have and don’t need a default GW. It looks like that pf rules doesn’t match/executed because there is no route.
              maybe we can swap to mail and only post the solution later .


              1. Hi Markus,

                I don’t think I understand. You don’t need a default route? Do you have specific routes installed for whatever destination(s) you need to reach?

                I’m also confused why you’re trying to “catch” traffic coming from the router. Are you trying to use pf to redirect that traffic to a non-default rdomain? Why aren’t you either configuring the daemon to use the right rdomain or using ‘route -T exec’?

                If you want to switch to email, my address is on the contact page (link at the top of the page).

  31. Hello,

    Great article and pretty much the only OpenBSD advanced routing article on the net because I keep ending back at yours. I try to accomplish something different with rdomains:

    Given 1 VM with 3 VNICs. The 3 VNICs are in the exact same network just on different IPs and I would like to use 3 different default routers on the 3 networks.

    ifconfig vio0 rdomain 1
    ifconfig vio0
    route -T1 add default
    route -T1 add 127/8
    ifconfig lo1 inet rdomain 1

    ifconfig vio1 rdomain 2
    ifconfig vio1
    route -T2 add default
    route -T2 add 127/8
    ifconfig lo2 inet rdomain 2

    ifconfig vio2 rdomain 3
    ifconfig vio2
    route -T3 add default
    route -T3 add 127/8
    ifconfig lo3 inet rdomain 3

    When adding the interfaces into the rdomains I run into the same problem as someone else in this thread:

    ifconfig vio2 rdomain 3
    ifconfig: SIOCSIFRDOMAIN: Invalid argument

    I don’t know if this is related to my issue but here is what happens when I define these route tables:

    I can do route -T3 exec ‘/usr/sbin/traceroute’ and the packet is going out the right gateway that is OK.

    However the problem is that the local network is not reachable through these tables.

    What I have tried separately one by one after flushing T3 clean:

    route -T3 add
    route -T3 add
    route -T3 add -inet -llinfo -link -static -iface vio2

    None of these are working the local network is not reachable directly through the vio2 link either. I would be running squid on this 3 rdomains which would use different gateways. The 3 squids only open their ports inside their tables. Any idea what is wrong with my setup?

    1. Hey FlashFloppy,

      Thanks for the feedback on the article.

      I tried to reproduce the scenario you’ve described and I’m unable to. I have a VM with two vNICs, each connected to the same virtual network. I do:

      ifconfig vmx0 rdomain 1
      ifconfig vmx0
      route -T1 add default

      ifconfig vmx1 rdomain 2
      ifconfig vmx1
      route -T2 add default

      I get no errors (including no SIOCSIFRDOMAIN warning). I can successfully ping a host on the subnet through either interface.

      route -T1 exec ping
      route -T2 exec ping

      I’m testing this on a fresh install of 6.5 on amd64; pf.conf is default.

  32. Yes you are correct. After running your command on OpenBSD 6.5 I get a proper routing table.

    ifconfig vio2 rdomain 2
    ifconfig vio2
    route -T2 add default

    route -n -T2 show
    Routing tables

    Destination Gateway Flags Refs Use Mtu Prio Iface
    default UGS 0 296 – 8 vio2
    172.16.7/24 UCn 1 0 – 4 vio2 link#3 UHLch 1 2 – 3 vio2 52:54:00:33:3b:10 UHLl 0 0 – 1 vio2 UHb 0 0 – 1 vio2

    Not like mine which was empty. So the issue was the first interface vio0 (which was the same ip range) and it’s main routing table interfered with mines.
    If I chose a completely different ip for the first interface then all the other routing tables will work fine.
    Thank you for your help.

    1. Ohh, interesting. I should’ve mentioned that in my test, I didn’t have a “main” interface. The VM only had vmx0 and vmx1 and I put both of them in an rdomain as shown.

      If I repeat the test with vmx0 left in rdomain 0 and vmx1 in rdomain 1, I still get good results. No errors, the routing tables populate as expected, and I can ping on the local subnet.

      Maybe there’s something fishy with the vio(4) driver?

Leave a Reply

Your email address will not be published. Required fields are marked *

Would you like to subscribe to email notification of new comments? You can also subscribe without commenting.