As a follow-up to my previous article on Port Channels titled "4 Types of Port Channels and When They're Used" I wanted to talk a bit about the long-standing rule that says you should always create your Etherchannel (EC) bundles with a number of links that works out to a power of two (ie, 2,4 or 8 links). That rule is less applicable today than it used to be.

Why Power of Two?

The short answer? In bundles that are not powers of two, when the hash algorithm is run, some links will always come up more than other links. This results in more traffic being put on some of the links and prevents an even spread.

The long answer? The switch hardware — and I'm talking specifically about the Catalyst 6500 in this post so that means the Policy Feature Card 3 (PFC3) found on the Sup720 — which is responsible for performing the EC hash uses a 3-bit hash. A 3-bit hash results in 8 possible values (called "buckets"). Now when an EC is formed, each of the links in the EC is assigned some number of buckets. In a two link EC, each link receives 4 buckets. In an eight link EC, each receives 1. When the PFC3 runs the hash on a packet, the output is a number that corresponds to the number of a bucket. Whichever link has that bucket is the one that will receive the packet.

In the case of a three link EC, since 8 is not divisible by 3, the buckets are not evenly distributed among the links.

Traffic Spread with 3-bit Hash
Bundle Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 Link 7 Link8
2 Links 50% 50%
3 Links 37.5% 37.5% 25%
4 Links 25% 25% 25% 25%
5 Links 25% 25% 25% 12.5% 12.5%
6 Links 25% 25% 12.5% 12.5% 12.5% 12.5%
7 Links 25% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5%
8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5%

This chart shows the traffic distribution breakdown for all possible link counts. Continuing with the 3 link example, you can see that the hash algorithm would chose the 3rd link less often than links 1 and 2 resulting in uneven utilization.

Bundles of Any Size

Since the size of the hash is baked into the hardware, there's no changing the "power of two" rule on the PFC3. However, with the PFC4 found on the Sup2T, the hash size has been changed to 8 bits which results in 256 buckets. This allows for a more fine-grained distribution of buckets across the available links. Again, not all links will receive an exactly even share of buckets (unless the link count is still a power of two) but the disparity between links is much lower.

Traffic Spread with 8-bit Hash
Bundle Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 Link 7 Link8
2 Links 50% 50%
3 Links 33.6% 33.2% 33.2%
4 Links 25% 25% 25% 25%
5 Links 20.4% 19.9% 19.9% 19.9% 19.9%
6 Links 16.8% 16.8% 16.8% 16.4% 16.4% 16.4%
7 Links 14.5% 14.5% 14.5% 14.5% 14% 14% 14%
8 Links 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5% 12.5%

So Which Hardware Does What Again?

On the Catalyst 6500, the PFC3/4 are found on these supervisors:

PFC Found on Supervisor Hash Size
PFC3 Sup32, Sup720 3-bit
PFC4 Sup2T 8-bit

So on the 6500, the Sup2T is able to more evenly distribute traffic across any sized EC.

And What About non-6500 Switches?

I don't have time to research all platforms, but I do know that the Nexus 7000 breaks down like this:

Linecard Hash Size
M1 3-bits
F1 8-bit

Since the supervisor on the Nexus 7000 does not handle any of the data plane functions, the hash size is entirely dependent on the linecard. I don't know specifically about the M2 or F2, but I would ASSuME they are the same as the M1/F1.


Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Cisco Systems.