As a follow-up to my previous article on Port Channels titled “4 Types of Port Channels and When They’re Used” I wanted to talk a bit about the long-standing rule that says you should always create your Etherchannel (EC) bundles with a number of links that works out to a power of two (ie, 2,4 or 8 links). That rule is less applicable today than it used to be.
Why Power of Two?
The short answer? In bundles that are not powers of two, when the hash algorithm is run, some links will always come up more than other links. This results in more traffic being put on some of the links and prevents an even spread.
The long answer? The switch hardware — and I’m talking specifically about the Catalyst 6500 in this post so that means the Policy Feature Card 3 (PFC3) found on the Sup720 — which is responsible for performing the EC hash uses a 3-bit hash. A 3-bit hash results in 8 possible values (called “buckets”). Now when an EC is formed, each of the links in the EC is assigned some number of buckets. In a two link EC, each link receives 4 buckets. In an eight link EC, each receives 1. When the PFC3 runs the hash on a packet, the output is a number that corresponds to the number of a bucket. Whichever link has that bucket is the one that will receive the packet.
In the case of a three link EC, since 8 is not divisible by 3, the buckets are not evenly distributed among the links.
| Bundle | Link 1 | Link 2 | Link 3 | Link 4 | Link 5 | Link 6 | Link 7 | Link8 |
| 2 Links | 50% | 50% | ||||||
| 3 Links | 37.5% | 37.5% | 25% | |||||
| 4 Links | 25% | 25% | 25% | 25% | ||||
| 5 Links | 25% | 25% | 25% | 12.5% | 12.5% | |||
| 6 Links | 25% | 25% | 12.5% | 12.5% | 12.5% | 12.5% | ||
| 7 Links | 25% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | |
| 8 Links | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% |
This chart shows the traffic distribution breakdown for all possible link counts. Continuing with the 3 link example, you can see that the hash algorithm would chose the 3rd link less often than links 1 and 2 resulting in uneven utilization.
Bundles of Any Size
Since the size of the hash is baked into the hardware, there’s no changing the “power of two” rule on the PFC3. However, with the PFC4 found on the Sup2T, the hash size has been changed to 8 bits which results in 256 buckets. This allows for a more fine-grained distribution of buckets across the available links. Again, not all links will receive an exactly even share of buckets (unless the link count is still a power of two) but the disparity between links is much lower.
| Bundle | Link 1 | Link 2 | Link 3 | Link 4 | Link 5 | Link 6 | Link 7 | Link8 |
| 2 Links | 50% | 50% | ||||||
| 3 Links | 33.6% | 33.2% | 33.2% | |||||
| 4 Links | 25% | 25% | 25% | 25% | ||||
| 5 Links | 20.4% | 19.9% | 19.9% | 19.9% | 19.9% | |||
| 6 Links | 16.8% | 16.8% | 16.8% | 16.4% | 16.4% | 16.4% | ||
| 7 Links | 14.5% | 14.5% | 14.5% | 14.5% | 14% | 14% | 14% | |
| 8 Links | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% |
So Which Hardware Does What Again?
On the Catalyst 6500, the PFC3/4 are found on these supervisors:
| PFC | Found on Supervisor | Hash Size |
| PFC3 | Sup32, Sup720 | 3-bit |
| PFC4 | Sup2T | 8-bit |
So on the 6500, the Sup2T is able to more evenly distribute traffic across any sized EC.
And What About non-6500 Switches?
I don’t have time to research all platforms, but I do know that the Nexus 7000 breaks down like this:
| Linecard | Hash Size |
| M1 | 3-bits |
| F1 | 8-bit |
Since the supervisor on the Nexus 7000 does not handle any of the data plane functions, the hash size is entirely dependent on the linecard. I don’t know specifically about the M2 or F2, but I would ASSuME they are the same as the M1/F1.
Doing Etherchannel Over 3, 5, 6, and 7 Link Bundles http://t.co/IwC51Zk8 (via @knight_joel)
Doing Etherchannel Over 3, 5, 6, and 7 Link Bundles | packetmischief.ca: http://t.co/UdfClebn
Good breakdown. Any ideas how the packets are hashed on a 3750 or 3750X?
Thanks Jason. I’ve looked pretty hard for info on the Cat 3k line but haven’t been able to find anything conclusive. If, and hopefully when, I find an answer, I will write a post about it.
Hello Joel,
I would be grateful if you can explain the Hash algorithm and indexing used in Catalyst 3750 switches. Because i noticed that on Catalyst 3560,3750,index number is always 0 for all interfaces. Please kindly note me on this.
Thank you
Hi Gayan,
I would love to give an update on the Cat 3k and 2k switches but I haven’t been able to find any information on the hash size they use. I’m pretty sure it’s going to be 3-bit. If and when I finally get that information, I will post it here.
When you say the index number is 0 for all interfaces, where are you seeing that?
Hi Joel,
This switch is a live switch on the net work, and show etherchannel port-chanel command out put is as follows.
_______3750G#show etherchannel port-channel
Channel-group listing:
———————-
Group: 1
———-
Port-channels in the group:
—————————
Port-channel: Po1
————
Age of the Port-channel = 49d:17h:02m:47s
Logical slot/port = 10/1 Number of ports = 2
GC = 0x00000000 HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = –
Ports in the Port-channel:
Index Load Port EC state No of bits
——+——+——+——————+———–
0 00 Gi1/0/1 On/FEC 0
0 00 Gi1/0/2 On/FEC 0
Time since last port bundled: 07d:23h:32m:43s Gi1/0/2
Time since last port Un-bundled: 07d:23h:54m:26s Gi1/0/2
Hey Gayan,
I see what you mean now. My understanding is that the index and also the load are always zeroes on the lower end switches. On the Cat 6k you’ll see numbers in both columns.
Have you ever used the “test etherchannel load-balance” command on the 3750? You give it a src and dst MAC or IP address and it will tell you which interface in the etherchannel the frames will be put on. That should help you get some more information as to where your flows are going.
Hello I was learning Load balancing in Etherchannel
I have one doubt regarding how RBH calculated.
SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.16 203.116.42.149
Computed RBH: 0x6
Would select Gi8/11 of Po1
How 0x6 value came??
Please help me :) Am very eager to know about this
Hi,
The RBH is the output from the etherchannel hashing algorithm and determines which physical port to use for an egress packet. If you use the “show etherchannel load-balance” command, you can see what information from the packet is fed into the hash algorithm. More info here:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020414
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804
Thanks for your amazing writing, it really helped me understand the whole thing, but I think in 3bit hashing table the 7 Links Bundle entries are miswritten, only the 1st link is 25%, and the next 6 Links must have the value of 12.5%.
Hi Erfaanism,
Right you are! Thanks for pointing that out. Thanks also for the feedback on the article :-)
Thanks for a nice write+up. Here is some info on 4500/3750:
https://supportforums.cisco.com/docs/DOC-17855
Great link. Thanks for sharing!
Thanks Joel, I was just asked this question and said without a doubt that uneven links would be uneven load. Came across your blog after a search and stand corrected. We do have Sup2Ts and therefore have a non-issue with uneven links. Thanks for the right up.
Thanks for commenting, Mark!