Doing Etherchannel Over 3, 5, 6, and 7 Link Bundles

As a follow-up to my previous article on Port Channels titled “4 Types of Port Channels and When They’re Used” I wanted to talk a bit about the long-standing rule that says you should always create your Etherchannel (EC) bundles with a number of links that works out to a power of two (ie, 2,4 or 8 links). That rule is less applicable today than it used to be.

Why Power of Two?

The short answer? In bundles that are not powers of two, when the hash algorithm is run, some links will always come up more than other links. This results in more traffic being put on some of the links and prevents an even spread.

The long answer? The switch hardware — and I’m talking specifically about the Catalyst 6500 in this post so that means the Policy Feature Card 3 (PFC3) found on the Sup720 — which is responsible for performing the EC hash uses a 3-bit hash. A 3-bit hash results in 8 possible values (called “buckets”). Now when an EC is formed, each of the links in the EC is assigned some number of buckets. In a two link EC, each link receives 4 buckets. In an eight link EC, each receives 1. When the PFC3 runs the hash on a packet, the output is a number that corresponds to the number of a bucket. Whichever link has that bucket is the one that will receive the packet.

In the case of a three link EC, since 8 is not divisible by 3, the buckets are not evenly distributed among the links.

Traffic Spread with 3-bit Hash
Bundle Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 Link 7 Link8
2 Links 50% 50%
3 Links  37.5% 37.5% 25%
4 Links 25%  25% 25% 25%
5 Links  25%  25% 25% 12.5% 12.5%
6 Links  25% 25%  12.5% 12.5% 12.5% 12.5%
7 Links  25%  12.5% 12.5% 12.5% 12.5% 12.5% 12.5%
8 Links  12.5% 12.5%  12.5%  12.5%  12.5%  12.5%  12.5%  12.5%

This chart shows the traffic distribution breakdown for all possible link counts. Continuing with the 3 link example, you can see that the hash algorithm would chose the 3rd link less often than links 1 and 2 resulting in uneven utilization.

Bundles of Any Size

Since the size of the hash is baked into the hardware, there’s no changing the “power of two” rule on the PFC3. However, with the PFC4 found on the Sup2T, the hash size has been changed to 8 bits which results in 256 buckets. This allows for a more fine-grained distribution of buckets across the available links. Again, not all links will receive an exactly even share of buckets (unless the link count is still a power of two) but the disparity between links is much lower.

Traffic Spread with 8-bit Hash
Bundle Link 1 Link 2 Link 3 Link 4 Link 5 Link 6 Link 7 Link8
2 Links 50% 50%
3 Links  33.6% 33.2%  33.2%
4 Links 25%  25% 25% 25%
5 Links  20.4%  19.9% 19.9% 19.9% 19.9%
6 Links  16.8% 16.8%  16.8% 16.4% 16.4% 16.4%
7 Links  14.5%  14.5% 14.5% 14.5% 14% 14% 14%
8 Links  12.5% 12.5%  12.5%  12.5%  12.5%  12.5%  12.5%  12.5%

So Which Hardware Does What Again?

On the Catalyst 6500, the PFC3/4 are found on these supervisors:

PFC Found on Supervisor Hash Size
PFC3 Sup32, Sup720 3-bit
PFC4  Sup2T 8-bit

So on the 6500, the Sup2T is able to more evenly distribute traffic across any sized EC.

And What About non-6500 Switches?

I don’t have time to research all platforms, but I do know that the Nexus 7000 breaks down like this:

Linecard Hash Size
M1 3-bits
 F1 8-bit

Since the supervisor on the Nexus 7000 does not handle any of the data plane functions, the hash size is entirely dependent on the linecard. I don’t know specifically about the M2 or F2, but I would ASSuME they are the same as the M1/F1.

Disclaimer: The opinions and information expressed in this blog article are my own and not those of Cisco Systems.

16 thoughts on “Doing Etherchannel Over 3, 5, 6, and 7 Link Bundles”

    1. Thanks Jason. I’ve looked pretty hard for info on the Cat 3k line but haven’t been able to find anything conclusive. If, and hopefully when, I find an answer, I will write a post about it.

  1. Hello Joel,
    I would be grateful if you can explain the Hash algorithm and indexing used in Catalyst 3750 switches. Because i noticed that on Catalyst 3560,3750,index number is always 0 for all interfaces. Please kindly note me on this.

    Thank you

    1. Hi Gayan,

      I would love to give an update on the Cat 3k and 2k switches but I haven’t been able to find any information on the hash size they use. I’m pretty sure it’s going to be 3-bit. If and when I finally get that information, I will post it here.

      When you say the index number is 0 for all interfaces, where are you seeing that?

      1. Hi Joel,

        This switch is a live switch on the net work, and show etherchannel port-chanel command out put is as follows.

        _______3750G#show etherchannel port-channel
        Channel-group listing:
        ———————-

        Group: 1
        ———-
        Port-channels in the group:
        —————————

        Port-channel: Po1
        ————

        Age of the Port-channel = 49d:17h:02m:47s
        Logical slot/port = 10/1 Number of ports = 2
        GC = 0x00000000 HotStandBy port = null
        Port state = Port-channel Ag-Inuse
        Protocol = –

        Ports in the Port-channel:

        Index Load Port EC state No of bits
        ——+——+——+——————+———–
        0 00 Gi1/0/1 On/FEC 0
        0 00 Gi1/0/2 On/FEC 0

        Time since last port bundled: 07d:23h:32m:43s Gi1/0/2
        Time since last port Un-bundled: 07d:23h:54m:26s Gi1/0/2

        1. Hey Gayan,

          I see what you mean now. My understanding is that the index and also the load are always zeroes on the lower end switches. On the Cat 6k you’ll see numbers in both columns.

          Have you ever used the “test etherchannel load-balance” command on the 3750? You give it a src and dst MAC or IP address and it will tell you which interface in the etherchannel the frames will be put on. That should help you get some more information as to where your flows are going.

  2. Hello I was learning Load balancing in Etherchannel
    I have one doubt regarding how RBH calculated.
    SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.16 203.116.42.149
    Computed RBH: 0x6
    Would select Gi8/11 of Po1
    How 0x6 value came??
    Please help me :) Am very eager to know about this

    1. Hi,

      The RBH is the output from the etherchannel hashing algorithm and determines which physical port to use for an egress packet. If you use the “show etherchannel load-balance” command, you can see what information from the packet is fed into the hash algorithm. More info here:

      http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020414

      http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804

  3. Thanks for your amazing writing, it really helped me understand the whole thing, but I think in 3bit hashing table the 7 Links Bundle entries are miswritten, only the 1st link is 25%, and the next 6 Links must have the value of 12.5%.

  4. Thanks Joel, I was just asked this question and said without a doubt that uneven links would be uneven load. Came across your blog after a search and stand corrected. We do have Sup2Ts and therefore have a non-issue with uneven links. Thanks for the right up.

Leave a Reply

Your email address will not be published. Required fields are marked *

Would you like to subscribe to email notification of new comments? You can also subscribe without commenting.