So… I’m a little embarrased to admit this but I only very recently found out that there are significant differences in how Virtual Port Channels (vPC) behave on the Nexus 5k vs the Nexus 7k when it comes to forming routing adjacencies over the vPC.
I’ve read the vPC Best Practice whitepaper and have often referred
others to it and also referred back to it myself from time to time. What I failed to realize is that I should’ve been taking the title of this paper more literally: it is 100% specific to the Nexus 7k. The behaviors the paper describes, particularly around the data plane loop prevention protections for packets crossing the vPC peer-link, are specific to the n7k and are not necessarily repeated on the n5k.
To that end, there are some topologies for Layer 3 peering over a vPC which are not supported on the n7k but are supported on the n5k. For example, peering a third-party Layer 3 device to the SVIs on the two n5ks in the vPC is supported.
The third-party device has an IP address on its port-channel interface and forms an Layer 3 adjacency/neighborship with each n5k. The n5ks also form an adjacency/neighborship with each other over the peer-link.
This is supported. And works! Everything is cool.
Until you start wondering what the little callouts in the diagram are hinting at and you take the time to read the config guide where this diagram is sourced from:
“[The diagram] illustrates the topology that is not recommended. In this topology, control protocol packets may be hashed by the port channel to the wrong Cisco Nexus 5500 Platform switch, which would then forward the control packets to the correct routing peer (126.96.36.199) in the picture. This topology is supported for unicast traffic but not for multicast traffic.”
What does “not recommended” mean?!?
It simply means BE CAUTIOUS! Know the implications of such a design. It does not mean Cisco TAC will not pick up the phone if you call in. It does not mean your n5ks will go up in smoke. It does not mean the world is ending. It means: stop and think.
Let’s break this down.
The design does not support routed multicast traffic.
- Are you running multicast traffic between these devices? Then this design is a non starter.
- Are you planning on running multicast traffic between these devices? Then this is still a non starter.
- Are you unsure if you might have to run multicast traffic between these devices one day? Wellllll… that’s a tough one that only you can decide on.
- If you’re confident that you’ll never have to run multicast traffic between these devices, then this design might be a viable one for your environment.
Now, what’s this business about packets hashing to the wrong n5k?
- The port-channel on the third-party device will use a hashing algorithm to determine which physical interface to put an outgoing packet onto.
- This algorithm is dumb and is not aware, nor does it take into account which n5k is connected on which physical interface. It also doesn’t know such things as which n5k is the OSPF Designated Router.
- It’s quite likely that the algorithm will put some packets on the “wrong” interface. Eg, if n5k-1 is the OSPF DR, the algorithm could put an OSPF packet that’s destined to the DR on the right-hand link in which case it would be received and processed by n5k-2.
- Since in this OSPF-based example the packet is a link-local multicast with a TTL of 1, n5k-2 will actually punt the packet to the CPU where it will be inspected, determined that the packet is actually destined to n5k-1, and will then be software switched out the peer-link to n5k-1.
- The reason this scenario is called out is because of the potential risk of increased CPU load on the n5ks and the resulting possibility of control plane instability.
- Once again, stop and think. Are you doing this for one or two third-party devices and CPU load on your n5ks is low? Probably not a big deal. Doing this for dozens of third-party devices with OSPF hello timers that have been tuned down to low values? Maybe that’ll be a problem.
I recommend reading the “Connecting to a Router in a vPC Topology” document linked below as well as the other sections of that document to fully understand vPC on the Nexus 5k. Unfortunately there is no vPC Best Practice doc for Nexus 5k like there is for the Nexus 7k.
- Hat tip to Adam Raffe for his excellent blog post on this topic: https://adamraffe.com/2013/03/08/l3-over-vpc-nexus-7000-vs-5000/
- Connecting to a Router in a vPC Topology (cisco.com)
- Layer 3 over vPC in NX-OS 7.3(0)N1(1)