Why EtherChannels should be used for FEX interfaces

I prefer to use port-channel interfaces for the fabric interfaces when connecting fabric extenders (FEXs). If a single interface in the bundle fails, it won’t remove the fabric extender interface – it simply reduces it’s bandwidth. This results in stable, predictable, redundant and resilient behavior. Let’s prove this point.

Using the same example environment that we’ve been using (see this article), FEX 100 is up, using Po100:

N5K(config-if)# show fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Po100 - Interface Up. State: Active
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-if)#

N5K(config-if)# sh run int po 100
version 4.0(1a)N2(1a)

interface port-channel100
  switchport mode fex-fabric
  fex associate 100

N5K(config-if)# sh run int eth1/19
version 4.0(1a)N2(1a)

interface Ethernet1/19
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

N5K(config-if)# sh run int eth1/20
version 4.0(1a)N2(1a)

interface Ethernet1/20
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

N5K(config-if)#

Everything matches thus far, so let’s continue by shutting down the physical interface Eth1/19:

N5K(config-if)# int eth1/19
N5K(config-if)# shut
N5K(config-if)#
N5K(config-if)#
N5K(config-if)# sh fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Po100 - Interface Up. State: Active
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
N5K(config-if)#

Nothing was taken down (as far as the FEX)! There’s no nifty show etherchannel command in NX-OS, which bums me a little (hopefully we’ll see it in the future?):

N5K# show ether?
            ^
% invalid command detected at '^' marker.
N5K# show e?
  environment  system environment information

N5K# show e

Not to fret, the show port-channel command is available in NX-OS:

N5K(config-if)# sh port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/1(P)    Eth1/2(P)
100   Po100(SU)   Eth      NONE      Eth1/19(D)   Eth1/20(P)
N5K(config-if)#

There are a couple of commands that can be helpful, but not as helpful to me in quickly looking at an EtherChannel on IOS:

N5K(config-if)# show int port-channel 100
port-channel100 is up
  Hardware is Port-Channel, address is 0011.2233.4455 (bia 0011.2233.4455)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is fex-fabric
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is off
  Switchport monitor is off
  Members in this channel: Eth1/19, Eth1/20
  Last clearing of "show interface" counters never
  5 minute input rate 385 bytes/sec, 0 packets/sec
  5 minute output rate 398 bytes/sec, 0 packets/sec
  Rx
    34186 input packets 4490 unicast packets 29550 multicast packets
    146 broadcast packets 246 jumbo packets 0 storm suppression packets
    31423540 bytes
    0 No buffer 0 runt 0 Overrun
    0 crc 0 Ignored 0 Bad etype drop
    0 Bad proto drop
  Tx
    52642 output packets 46806 multicast packets
    129 broadcast packets 1340 jumbo packets
    15879684 bytes
    0 output CRC 0 ecc
    0 underrun 0 if down drop     0 output error 0 collision 0 deferred
    0 late collision 0 lost carrier 0 no carrier
    0 babble
    0 Rx pause 0 Tx pause 0 reset

N5K(config-if)# show int port-channel 100 brief

--------------------------------------------------------------------------------
Port-channel VLAN  Type Mode   Status  Reason                    Speed  Protocol
Interface
--------------------------------------------------------------------------------
Po100        1     eth  fabric up      none                         10G(D)  none
N5K(config-if)#

Now let’s bring Eth1/19 back up – nothing happens (except we have additional bandwidth)!

N5K# show int port-channel 100
port-channel100 is up
  Hardware is Port-Channel, address is 0011.2233.4455 (bia 0011.2233.4455)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is fex-fabric
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is off
  Switchport monitor is off
  Members in this channel: Eth1/19, Eth1/20
  Last clearing of "show interface" counters never
  5 minute input rate 279 bytes/sec, 0 packets/sec
  5 minute output rate 280 bytes/sec, 0 packets/sec
  Rx
    34370 input packets 4599 unicast packets 29625 multicast packets
    146 broadcast packets 246 jumbo packets 0 storm suppression packets
    31484185 bytes
    0 No buffer 0 runt 0 Overrun
    0 crc 0 Ignored 0 Bad etype drop
    0 Bad proto drop
  Tx
    52829 output packets 46884 multicast packets
    129 broadcast packets 1340 jumbo packets
    15999620 bytes
    0 output CRC 0 ecc
    0 underrun 0 if down drop     0 output error 0 collision 0 deferred
    0 late collision 0 lost carrier 0 no carrier
    0 babble
    0 Rx pause 0 Tx pause 0 reset

N5K# conf t
N5K(config)# int eth1/19
N5K(config-if)# no shut
N5K(config-if)# sh int port- 100
port-channel100 is up
  Hardware is Port-Channel, address is 0011.2233.4455 (bia 0011.2233.4455)
  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is fex-fabric
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is off
  Switchport monitor is off
  Members in this channel: Eth1/19, Eth1/20
  Last clearing of "show interface" counters never
  5 minute input rate 274 bytes/sec, 0 packets/sec
  5 minute output rate 280 bytes/sec, 0 packets/sec
  Rx
    34382 input packets 4606 unicast packets 29630 multicast packets
    146 broadcast packets 246 jumbo packets 0 storm suppression packets
    31488165 bytes
    0 No buffer 0 runt 0 Overrun
    0 crc 0 Ignored 0 Bad etype drop
    0 Bad proto drop
  Tx
    52841 output packets 46889 multicast packets
    129 broadcast packets 1340 jumbo packets
    16007492 bytes
    0 output CRC 0 ecc
    0 underrun 0 if down drop     0 output error 0 collision 0 deferred
    0 late collision 0 lost carrier 0 no carrier
    0 babble
    0 Rx pause 0 Tx pause 0 reset

N5K(config-if)#

The FEX never flapped (the state stayed the same), however the bandwidth changed (20Gbps down to 10Gbps and then back up to 20Gbps). This is great! I’d much rather lose some bandwidth than lose a portion of my host ports on the FEX (as is the case when using static pinning when a fabric interface goes down).

One question that I have as far as how the Nexus ASICs handle changes in the EtherChannel bundle is whether there is any traffic loss while the bundle re-calculates the table for the EtherChannel hashing algorithm. This I’m not sure about, but would like to know. If packets are buffered while the table is being recalculated, it would work in an effort to provide a lossless transition upon a failure of the bundle member, however if it drops packets during this momentary period, that would be undesirable.

I definitely prefer to use EtherChannels for fabric interfaces whenever possible – hopefully the above shows why.

The FEX information is cached until it’s re-assigned to a different FEX Continuing our journey into the world of FEXs on the...
What happens when the wrong serial number is entered for the FEX? Continuing on the journey of FEXs on the Nexus platform,...
Experimenting with Static Pinning What happens when more links are associated with an FEX...
FEX configs are retained The N5k maintains the config of the FEXs, even when...
Methods for connecting FEXs If you’re wondering what an FEX is, check out this...

Related posts brought to you by Yet Another Related Posts Plugin.

Tags: 2148T, associate, bundle, channel-group, cisco, config, configuration, etherchannel, example, fabric interface, fex, n2148t, n5010, n5020, nexus, nexus 2000, nexus 2148T, nexus 5000, nexus 5010, nexus 5020, nx-os, port-channel, training, tutorial

This entry was posted on Monday, February 8th, 2010 at 11:25 am and is filed under Cisco Nexus, Networking. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

CCIEZone.com

Why EtherChannels should be used for FEX interfaces

Leave a Reply