Experimenting with Static Pinning

What happens when more links are associated with an FEX than are permitted in the max-links parameter? Let’s find out…

Let’s go back to FEX 100 (in our example environment):

!
fex 100
  pinning max-links 1
  description First_Nexus_2148T
  serial PRIVATE_SERIAL
  type "Nexus 2148T"
!

This FEX is configured to allow 1 link to be pinned. What happens if we associate multiple (more than one) interfaces to this FEX?

N5K(config)# int ether 1/19-20
N5K(config-if-range)# fex associate 100
N5K(config-if-range)# 2010 Jan 27 15:07:44 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL)

N5K(config-if-range)#
N5K(config-if-range)# show fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/19
  Fabric interface state:
    Po100 - Interface Down. State: Configured
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-if-range)#

Hmm – this obviously worked (brought up the 2148T, shows both interfaces as active). Let’s confirm using some nifty show commands:

N5K(config-if-range)# show interface fex-fabric
     Fabric      Fabric       Fex                FEX
Fex  Port      Port State    Uplink    Model         Serial
---------------------------------------------------------------
100   Eth1/19        Active     1    N2K-C2148T-1GE  PRIVATE_SERIAL
100   Eth1/20        Active     2    N2K-C2148T-1GE  PRIVATE_SERIAL
N5K(config-if-range)# show int ether 1/19 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if-range)# show int ether 1/20 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if-range)#

I see a problem – there are no interfaces associated with either of the physical interfaces. Let’s try removing one of the links from the FEX and see if this resolves itself. We’ve dis-associated Eth1/20 from FEX 100:

N5K(config-if)# show interface fex-fabric
     Fabric      Fabric       Fex                FEX
Fex  Port      Port State    Uplink    Model         Serial
---------------------------------------------------------------
100   Eth1/19        Active     1    N2K-C2148T-1GE  PRIVATE_SERIAL
100   Eth1/20    Discovered     2    N2K-C2148T-1GE  PRIVATE_SERIAL
N5K(config-if)#
N5K(config-if)#
N5K(config-if)# show interface fex-fabric
     Fabric      Fabric       Fex                FEX
Fex  Port      Port State    Uplink    Model         Serial
---------------------------------------------------------------
100   Eth1/19        Active     1    N2K-C2148T-1GE  PRIVATE_SERIAL
100   Eth1/20    Discovered     2    N2K-C2148T-1GE  PRIVATE_SERIAL
N5K(config-if)#
N5K(config-if)#
N5K(config-if)# show int ether 1/19 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)# show int ether 1/20 fex-intf
N5K(config-if)#

Note that Eth1/20 is listed as Discovered – this is because it sees the N2148T (the mode is still fex-fabric for Eth1/20), so it’s pretty much indicating that it sees something there, but it’s not Active, so it’s not actively participating in the FEX.

Let’s dig deeper:

N5K(config-if)# show fex 100 detail
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/19
  Fabric interface state:
    Po100 - Interface Down. State: Configured
    Eth1/19 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1  Down       Po100       Po100
       Eth100/1/2  Down       Po100       Po100
       Eth100/1/3  Down       Po100       Po100
       Eth100/1/4  Down       Po100       Po100
       Eth100/1/5  Down       Po100       Po100
       Eth100/1/6  Down       Po100       Po100
       Eth100/1/7  Down       Po100       Po100
       Eth100/1/8  Down       Po100       Po100
       Eth100/1/9  Down       Po100       Po100
      Eth100/1/10  Down       Po100       Po100
      Eth100/1/11  Down       Po100       Po100
      Eth100/1/12  Down       Po100       Po100
      Eth100/1/13  Down       Po100       Po100
      Eth100/1/14  Down       Po100       Po100
      Eth100/1/15  Down       Po100       Po100
      Eth100/1/16  Down       Po100       Po100
      Eth100/1/17  Down       Po100       Po100
      Eth100/1/18  Down       Po100       Po100
      Eth100/1/19  Down       Po100       Po100
      Eth100/1/20  Down       Po100       Po100
      Eth100/1/21  Down       Po100       Po100
      Eth100/1/22  Down       Po100       Po100
      Eth100/1/23  Down       Po100       Po100
      Eth100/1/24  Down       Po100       Po100
      Eth100/1/25  Down       Po100       Po100
      Eth100/1/26  Down       Po100       Po100
      Eth100/1/27  Down       Po100       Po100
      Eth100/1/28  Down       Po100       Po100
      Eth100/1/29  Down       Po100       Po100
      Eth100/1/30  Down       Po100       Po100
      Eth100/1/31  Down       Po100       Po100
      Eth100/1/32  Down       Po100       Po100
      Eth100/1/33  Down       Po100       Po100
      Eth100/1/34  Down       Po100       Po100
      Eth100/1/35  Down       Po100       Po100
      Eth100/1/36  Down       Po100       Po100
      Eth100/1/37  Down       Po100       Po100
      Eth100/1/38  Down       Po100       Po100
      Eth100/1/39  Down       Po100       Po100
      Eth100/1/40  Down       Po100       Po100
      Eth100/1/41  Down       Po100       Po100
      Eth100/1/42  Down       Po100       Po100
      Eth100/1/43  Down       Po100       Po100
      Eth100/1/44  Down       Po100       Po100
      Eth100/1/45  Down       Po100       Po100
      Eth100/1/46  Down       Po100       Po100
      Eth100/1/47  Down       Po100       Po100
      Eth100/1/48  Down       Po100       Po100
Logs:
...
[01/27/2010 15:06:50.6667] Module Offline
[01/27/2010 15:07:43.409418] Module register received
[01/27/2010 15:07:43.410369] Registration response sent
[01/27/2010 15:07:43.495794] Module Online Sequence
[01/27/2010 15:07:44.822818] Module Online
N5K(config-if)#

Okay, now things are coming together. Even the Po100 interface isn’t up, the FEX sees this as the one interface and ties everything to it. Let’s validate this in the config:

N5K(config-if)# sh run int po100
version 4.0(1a)N2(1a)

interface port-channel100
  switchport mode fex-fabric
  fex associate 100

N5K(config-if)#

Now let’s remove it:

N5K(config-if)# int po100
N5K(config-if)# no fex associate
N5K(config-if)#

Now verify:

N5K(config-if)# show fex 100 detail
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/19
  Fabric interface state:
    Eth1/19 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1    Up     Eth1/19     Eth1/19
       Eth100/1/2  Down     Eth1/19     Eth1/19
       Eth100/1/3  Down     Eth1/19     Eth1/19
       Eth100/1/4  Down     Eth1/19     Eth1/19
       Eth100/1/5  Down     Eth1/19     Eth1/19
       Eth100/1/6  Down     Eth1/19     Eth1/19
       Eth100/1/7  Down     Eth1/19     Eth1/19
       Eth100/1/8  Down     Eth1/19     Eth1/19
       Eth100/1/9  Down     Eth1/19     Eth1/19
      Eth100/1/10  Down     Eth1/19     Eth1/19
      Eth100/1/11  Down     Eth1/19     Eth1/19
      Eth100/1/12  Down     Eth1/19     Eth1/19
      Eth100/1/13  Down     Eth1/19     Eth1/19
      Eth100/1/14  Down     Eth1/19     Eth1/19
      Eth100/1/15  Down     Eth1/19     Eth1/19
      Eth100/1/16  Down     Eth1/19     Eth1/19
      Eth100/1/17  Down     Eth1/19     Eth1/19
      Eth100/1/18  Down     Eth1/19     Eth1/19
      Eth100/1/19  Down     Eth1/19     Eth1/19
      Eth100/1/20  Down     Eth1/19     Eth1/19
      Eth100/1/21  Down     Eth1/19     Eth1/19
      Eth100/1/22  Down     Eth1/19     Eth1/19
      Eth100/1/23  Down     Eth1/19     Eth1/19
      Eth100/1/24  Down     Eth1/19     Eth1/19
      Eth100/1/25  Down     Eth1/19     Eth1/19
      Eth100/1/26  Down     Eth1/19     Eth1/19
      Eth100/1/27  Down     Eth1/19     Eth1/19
      Eth100/1/28  Down     Eth1/19     Eth1/19
      Eth100/1/29  Down     Eth1/19     Eth1/19
      Eth100/1/30  Down     Eth1/19     Eth1/19
      Eth100/1/31  Down     Eth1/19     Eth1/19
      Eth100/1/32  Down     Eth1/19     Eth1/19
      Eth100/1/33  Down     Eth1/19     Eth1/19
      Eth100/1/34  Down     Eth1/19     Eth1/19
      Eth100/1/35  Down     Eth1/19     Eth1/19
      Eth100/1/36  Down     Eth1/19     Eth1/19
      Eth100/1/37  Down     Eth1/19     Eth1/19
      Eth100/1/38  Down     Eth1/19     Eth1/19
      Eth100/1/39  Down     Eth1/19     Eth1/19
      Eth100/1/40  Down     Eth1/19     Eth1/19
      Eth100/1/41  Down     Eth1/19     Eth1/19
      Eth100/1/42  Down     Eth1/19     Eth1/19
      Eth100/1/43  Down     Eth1/19     Eth1/19
      Eth100/1/44  Down     Eth1/19     Eth1/19
      Eth100/1/45  Down     Eth1/19     Eth1/19
      Eth100/1/46  Down     Eth1/19     Eth1/19
      Eth100/1/47  Down     Eth1/19     Eth1/19
      Eth100/1/48  Down     Eth1/19     Eth1/19
Logs:
...
N5K(config-if)#

Bingo! Let’s check out some of the other commands we were looking at:

N5K(config-if)# show int ether 1/19 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/19         Eth100/1/48   Eth100/1/47   Eth100/1/46   Eth100/1/45
                 Eth100/1/44   Eth100/1/43   Eth100/1/42   Eth100/1/41
                 Eth100/1/40   Eth100/1/39   Eth100/1/38   Eth100/1/37
                 Eth100/1/36   Eth100/1/35   Eth100/1/34   Eth100/1/33
                 Eth100/1/32   Eth100/1/31   Eth100/1/30   Eth100/1/29
                 Eth100/1/28   Eth100/1/27   Eth100/1/26   Eth100/1/25
                 Eth100/1/24   Eth100/1/23   Eth100/1/22   Eth100/1/21
                 Eth100/1/20   Eth100/1/19   Eth100/1/18   Eth100/1/17
                 Eth100/1/16   Eth100/1/15   Eth100/1/14   Eth100/1/13
                 Eth100/1/12   Eth100/1/11   Eth100/1/10   Eth100/1/9
                 Eth100/1/8    Eth100/1/7    Eth100/1/6    Eth100/1/5
                 Eth100/1/4    Eth100/1/3    Eth100/1/2    Eth100/1/1
N5K(config-if)#

Now what happens if we exceed the number of links in the FEX? Let’s find out and associate Eth1/20 back with FEX 100. Here’s the output after we’ve associated Eth1/20 with FEX 100:

N5K(config-if)# show fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/19
  Fabric interface state:
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-if)#
N5K(config-if)# show int ether 1/19 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/19         Eth100/1/48   Eth100/1/47   Eth100/1/46   Eth100/1/45
                 Eth100/1/44   Eth100/1/43   Eth100/1/42   Eth100/1/41
                 Eth100/1/40   Eth100/1/39   Eth100/1/38   Eth100/1/37
                 Eth100/1/36   Eth100/1/35   Eth100/1/34   Eth100/1/33
                 Eth100/1/32   Eth100/1/31   Eth100/1/30   Eth100/1/29
                 Eth100/1/28   Eth100/1/27   Eth100/1/26   Eth100/1/25
                 Eth100/1/24   Eth100/1/23   Eth100/1/22   Eth100/1/21
                 Eth100/1/20   Eth100/1/19   Eth100/1/18   Eth100/1/17
                 Eth100/1/16   Eth100/1/15   Eth100/1/14   Eth100/1/13
                 Eth100/1/12   Eth100/1/11   Eth100/1/10   Eth100/1/9
                 Eth100/1/8    Eth100/1/7    Eth100/1/6    Eth100/1/5
                 Eth100/1/4    Eth100/1/3    Eth100/1/2    Eth100/1/1
N5K(config-if)# show int ether 1/20 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)#

This is good – they’re both added to the FEX, but no ports are assigned to Eth1/20 (the port that exceeds the pinning max-links parameter). Let’s play around a little and see if we can re-distribute the interfaces (maybe we can trick NX-OS into spreading them across both, even though we only want one?):

N5K# fex pinning redistribute 100
No redistribution needed on FEX 100
N5K#

No tricking it today!

How about if we shutdown the first interface (Eth1/19) – will it down the FEX or will it automatically spread the interfaces to the remaining interface (Eth1/20)?

N5K# conf t
N5K(config)# int eth1/19
N5K(config-if)# shut
N5K(config-if)# sh fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
N5K(config-if)# sh int eth1/19 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)# sh int eth1/20 fex-intf
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)#

No – no-go. NX-OS uses the first interface defined (whether or not it’s up). It’s statically pinned (hence the name) to the group of ports (which is determined by the pinning max-links parameter).

I have a question – is the ordering based on configuration order or does it re-order them based on the interface name (ie. Port-Channel versus Ethernet)?

Let’s find out – let’s add back in the Po100 interface:

N5K(config-if)# int po100
N5K(config-if)# fex associate 100
N5K(config-if)# sh fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Po100 - Interface Down. State: Configured
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
N5K(config-if)#

Interesting – let’s bring back up Eth1/19 now and see what happens:

N5K(config-if)# int eth1/19
N5K(config-if)# no shut
N5K(config-if)# sh fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Po100 - Interface Down. State: Configured
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-if)# sh int eth 1/19 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)#

Nothing. So, the ordering of the interfaces is based on the name. Port-Channel interfaces take precedence over Ethernet interfaces, for some reason – maybe they’re sorted alphabetically in reverse? As can be seen above, there are no interfaces tied to Eth1/19 (and definitely won’t be on Eth1/20). Let’s confirm:

N5K(config-if)# show int eth1/20 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)# show int po100 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)# show int eth1/19 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-if)#

So because the first interface is down, NX-OS doesn’t use any remaining interfaces (because the pinning takes the first interface shown in the list of Fabric interfaces, not the first available (up/up) interface in the list). This behaves more like a static list, in that there’s no searching down the list.

Let’s adjust the max-links and see what happens:

N5K(config-fex)# pinning max-links 2
Error: FEX has port-channel. max-links should be one
N5K(config-fex)#

Oh – I forgot about this! Port-Channels can only be used with a max-link of 1. We’ll now remove the Po100 interface then try it again:

N5K(config-fex)# int po100
N5K(config-if)# no fex associate
N5K(config-if)# fex 100
N5K(config-fex)# sho fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Keepalive pending for 1 intervals
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-fex)# show int eth1/19 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/19         Eth100/1/48   Eth100/1/47   Eth100/1/46   Eth100/1/45
                 Eth100/1/44   Eth100/1/43   Eth100/1/42   Eth100/1/41
                 Eth100/1/40   Eth100/1/39   Eth100/1/38   Eth100/1/37
                 Eth100/1/36   Eth100/1/35   Eth100/1/34   Eth100/1/33
                 Eth100/1/32   Eth100/1/31   Eth100/1/30   Eth100/1/29
                 Eth100/1/28   Eth100/1/27   Eth100/1/26   Eth100/1/25
                 Eth100/1/24   Eth100/1/23   Eth100/1/22   Eth100/1/21
                 Eth100/1/20   Eth100/1/19   Eth100/1/18   Eth100/1/17
                 Eth100/1/16   Eth100/1/15   Eth100/1/14   Eth100/1/13
                 Eth100/1/12   Eth100/1/11   Eth100/1/10   Eth100/1/9
                 Eth100/1/8    Eth100/1/7    Eth100/1/6    Eth100/1/5
                 Eth100/1/4    Eth100/1/3    Eth100/1/2    Eth100/1/1
N5K(config-fex)#

Eth1/19 is back up-and-running. Now shutdown Eth1/19 and adjust the max-links parameter:

N5K(config-fex)# int eth1/19
N5K(config-if)# shut
N5K(config-if)#
N5K(config-if)#
N5K(config-if)# fex 100
N5K(config-fex)# pinning max-links 2
Change in Max-links will cause traffic disruption.
N5K(config-fex)#
N5K(config-fex)#
N5K(config-fex)# show fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
N5K(config-fex)# show int eth1/20 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/20         Eth100/1/48   Eth100/1/47   Eth100/1/46   Eth100/1/45
                 Eth100/1/44   Eth100/1/43   Eth100/1/42   Eth100/1/41
                 Eth100/1/40   Eth100/1/39   Eth100/1/38   Eth100/1/37
                 Eth100/1/36   Eth100/1/35   Eth100/1/34   Eth100/1/33
                 Eth100/1/32   Eth100/1/31   Eth100/1/30   Eth100/1/29
                 Eth100/1/28   Eth100/1/27   Eth100/1/26   Eth100/1/25
N5K(config-fex)# show int eth1/19 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
N5K(config-fex)#

This proves what is said in the Nexus 2000 NX-OS config guide that if a fex interface is down, the associated interfaces on the 2148T (fabric extender) will remain down until the interface is resolved (restored to service). See http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_4_1/Cisco_Nexus_2000_Series_Fabric_Extender_Software_Configuration_Guide_chapter2.html#con_1203877 for more info (look at the note).

Okay, well let’s go ahead and bring up Eth1/19 now:

N5K(config-if)# sh int eth1/19 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/19         Eth100/1/24   Eth100/1/23   Eth100/1/22   Eth100/1/21
                 Eth100/1/20   Eth100/1/19   Eth100/1/18   Eth100/1/17
                 Eth100/1/16   Eth100/1/15   Eth100/1/14   Eth100/1/13
                 Eth100/1/12   Eth100/1/11   Eth100/1/10   Eth100/1/9
                 Eth100/1/8    Eth100/1/7    Eth100/1/6    Eth100/1/5
                 Eth100/1/4    Eth100/1/3    Eth100/1/2    Eth100/1/1
N5K(config-if)# sh int eth1/20 fex
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Eth1/20         Eth100/1/48   Eth100/1/47   Eth100/1/46   Eth100/1/45
                 Eth100/1/44   Eth100/1/43   Eth100/1/42   Eth100/1/41
                 Eth100/1/40   Eth100/1/39   Eth100/1/38   Eth100/1/37
                 Eth100/1/36   Eth100/1/35   Eth100/1/34   Eth100/1/33
                 Eth100/1/32   Eth100/1/31   Eth100/1/30   Eth100/1/29
                 Eth100/1/28   Eth100/1/27   Eth100/1/26   Eth100/1/25
N5K(config-if)# sho fex 100
FEX: 100 Description: First_Nexus_2148T   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Up. State: Active
    Eth1/20 - Interface Up. State: Active
N5K(config-if)#

The note in the config guide is why NX-OS doesn’t failover between remaining fabric interfaces (because the interfaces are pinned to each interface in the order shown in the output of the show fex command).

One last item to mention here is that if two links are specified and only one link exists, only half of the FEX interfaces will work!

Eth1/19 and Eth1/20 are used in the following example. Eth1/19 has been disassociated from FEX 100, and here’s the output:

N5K(config-if)# sh fex 100 detail
FEX: 100 Description: POD16 FEX 100   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/20 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1  Down     Eth1/19        None
       Eth100/1/2  Down     Eth1/19        None
       Eth100/1/3  Down     Eth1/19        None
       Eth100/1/4  Down     Eth1/19        None
       Eth100/1/5  Down     Eth1/19        None
       Eth100/1/6  Down     Eth1/19        None
       Eth100/1/7  Down     Eth1/19        None
       Eth100/1/8  Down     Eth1/19        None
       Eth100/1/9  Down     Eth1/19        None
      Eth100/1/10  Down     Eth1/19        None
      Eth100/1/11  Down     Eth1/19        None
      Eth100/1/12  Down     Eth1/19        None
      Eth100/1/13  Down     Eth1/19        None
      Eth100/1/14  Down     Eth1/19        None
      Eth100/1/15  Down     Eth1/19        None
      Eth100/1/16  Down     Eth1/19        None
      Eth100/1/17  Down     Eth1/19        None
      Eth100/1/18  Down     Eth1/19        None
      Eth100/1/19  Down     Eth1/19        None
      Eth100/1/20  Down     Eth1/19        None
      Eth100/1/21  Down     Eth1/19        None
      Eth100/1/22  Down     Eth1/19        None
      Eth100/1/23  Down     Eth1/19        None
      Eth100/1/24  Down     Eth1/19        None
      Eth100/1/25  Down     Eth1/20     Eth1/20
      Eth100/1/26  Down     Eth1/20     Eth1/20
      Eth100/1/27  Down     Eth1/20     Eth1/20
      Eth100/1/28  Down     Eth1/20     Eth1/20
      Eth100/1/29  Down     Eth1/20     Eth1/20
      Eth100/1/30  Down     Eth1/20     Eth1/20
      Eth100/1/31  Down     Eth1/20     Eth1/20
      Eth100/1/32  Down     Eth1/20     Eth1/20
      Eth100/1/33  Down     Eth1/20     Eth1/20
      Eth100/1/34  Down     Eth1/20     Eth1/20
      Eth100/1/35  Down     Eth1/20     Eth1/20
      Eth100/1/36  Down     Eth1/20     Eth1/20
      Eth100/1/37  Down     Eth1/20     Eth1/20
      Eth100/1/38  Down     Eth1/20     Eth1/20
      Eth100/1/39  Down     Eth1/20     Eth1/20
      Eth100/1/40  Down     Eth1/20     Eth1/20
      Eth100/1/41  Down     Eth1/20     Eth1/20
      Eth100/1/42  Down     Eth1/20     Eth1/20
      Eth100/1/43  Down     Eth1/20     Eth1/20
      Eth100/1/44  Down     Eth1/20     Eth1/20
      Eth100/1/45  Down     Eth1/20     Eth1/20
      Eth100/1/46  Down     Eth1/20     Eth1/20
      Eth100/1/47  Down     Eth1/20     Eth1/20
      Eth100/1/48  Down     Eth1/20     Eth1/20
Logs:

Notice how Eth100/1-24 don’t have any primary fabric interface!

Let’s try redistributing the interfaces:

N5K(config-if)# end
N5K# fex pinning redistribute 100
Redistributing all ports on FEX 100. This command will cause traffic disruption.
N5K#

Now let’s look at the details of FEX 100:

N5K# show fex 100 detail
FEX: 100 Description: POD16 FEX 100   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/20 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1    Up     Eth1/20     Eth1/20
       Eth100/1/2  Down     Eth1/20     Eth1/20
       Eth100/1/3  Down     Eth1/20     Eth1/20
       Eth100/1/4  Down     Eth1/20     Eth1/20
       Eth100/1/5  Down     Eth1/20     Eth1/20
       Eth100/1/6  Down     Eth1/20     Eth1/20
       Eth100/1/7  Down     Eth1/20     Eth1/20
       Eth100/1/8  Down     Eth1/20     Eth1/20
       Eth100/1/9  Down     Eth1/20     Eth1/20
      Eth100/1/10  Down     Eth1/20     Eth1/20
      Eth100/1/11  Down     Eth1/20     Eth1/20
      Eth100/1/12  Down     Eth1/20     Eth1/20
      Eth100/1/13  Down     Eth1/20     Eth1/20
      Eth100/1/14  Down     Eth1/20     Eth1/20
      Eth100/1/15  Down     Eth1/20     Eth1/20
      Eth100/1/16  Down     Eth1/20     Eth1/20
      Eth100/1/17  Down     Eth1/20     Eth1/20
      Eth100/1/18  Down     Eth1/20     Eth1/20
      Eth100/1/19  Down     Eth1/20     Eth1/20
      Eth100/1/20  Down     Eth1/20     Eth1/20
      Eth100/1/21  Down     Eth1/20     Eth1/20
      Eth100/1/22  Down     Eth1/20     Eth1/20
      Eth100/1/23  Down     Eth1/20     Eth1/20
      Eth100/1/24  Down     Eth1/20     Eth1/20
      Eth100/1/25  Down        None        None
      Eth100/1/26  Down        None        None
      Eth100/1/27  Down        None        None
      Eth100/1/28  Down        None        None
      Eth100/1/29  Down        None        None
      Eth100/1/30  Down        None        None
      Eth100/1/31  Down        None        None
      Eth100/1/32  Down        None        None
      Eth100/1/33  Down        None        None
      Eth100/1/34  Down        None        None
      Eth100/1/35  Down        None        None
      Eth100/1/36  Down        None        None
      Eth100/1/37  Down        None        None
      Eth100/1/38  Down        None        None
      Eth100/1/39  Down        None        None
      Eth100/1/40  Down        None        None
      Eth100/1/41  Down        None        None
      Eth100/1/42  Down        None        None
      Eth100/1/43  Down        None        None
      Eth100/1/44  Down        None        None
      Eth100/1/45  Down        None        None
      Eth100/1/46  Down        None        None
      Eth100/1/47  Down        None        None
      Eth100/1/48  Down        None        None
Logs:

Hmm – it flip-flopped things around, so that Eth100/1/1-24 are tied to Eth1/20, but Eth100/1/25-48 are not associated with any physical interface. Bummer. We could change the max-links to 1, then all of the FEX interfaces (Eth100/1/1-48) would be tied to Eth1/20, or we could re-associate Eth1/19. This is what we’ll do – add Eth1/19 back to the FEX:

N5K(config)# int eth1/19
N5K(config-if)# fex associate 100
N5K(config-if)#
N5K(config-if)#
N5K(config-if)#
N5K(config-if)#
N5K(config-if)# sh fex 100 detail
FEX: 100 Description: POD16 FEX 100   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1    Up     Eth1/20     Eth1/20
       Eth100/1/2  Down     Eth1/20     Eth1/20
       Eth100/1/3  Down     Eth1/20     Eth1/20
       Eth100/1/4  Down     Eth1/20     Eth1/20
       Eth100/1/5  Down     Eth1/20     Eth1/20
       Eth100/1/6  Down     Eth1/20     Eth1/20
       Eth100/1/7  Down     Eth1/20     Eth1/20
       Eth100/1/8  Down     Eth1/20     Eth1/20
       Eth100/1/9  Down     Eth1/20     Eth1/20
      Eth100/1/10  Down     Eth1/20     Eth1/20
      Eth100/1/11  Down     Eth1/20     Eth1/20
      Eth100/1/12  Down     Eth1/20     Eth1/20
      Eth100/1/13  Down     Eth1/20     Eth1/20
      Eth100/1/14  Down     Eth1/20     Eth1/20
      Eth100/1/15  Down     Eth1/20     Eth1/20
      Eth100/1/16  Down     Eth1/20     Eth1/20
      Eth100/1/17  Down     Eth1/20     Eth1/20
      Eth100/1/18  Down     Eth1/20     Eth1/20
      Eth100/1/19  Down     Eth1/20     Eth1/20
      Eth100/1/20  Down     Eth1/20     Eth1/20
      Eth100/1/21  Down     Eth1/20     Eth1/20
      Eth100/1/22  Down     Eth1/20     Eth1/20
      Eth100/1/23  Down     Eth1/20     Eth1/20
      Eth100/1/24  Down     Eth1/20     Eth1/20
      Eth100/1/25  Down        None     Eth1/19
      Eth100/1/26  Down        None     Eth1/19
      Eth100/1/27  Down        None     Eth1/19
      Eth100/1/28  Down        None     Eth1/19
      Eth100/1/29  Down        None     Eth1/19
      Eth100/1/30  Down        None     Eth1/19
      Eth100/1/31  Down        None     Eth1/19
      Eth100/1/32  Down        None     Eth1/19
      Eth100/1/33  Down        None     Eth1/19
      Eth100/1/34  Down        None     Eth1/19
      Eth100/1/35  Down        None     Eth1/19
      Eth100/1/36  Down        None     Eth1/19
      Eth100/1/37  Down        None     Eth1/19
      Eth100/1/38  Down        None     Eth1/19
      Eth100/1/39  Down        None     Eth1/19
      Eth100/1/40  Down        None     Eth1/19
      Eth100/1/41  Down        None     Eth1/19
      Eth100/1/42  Down        None     Eth1/19
      Eth100/1/43  Down        None     Eth1/19
      Eth100/1/44  Down        None     Eth1/19
      Eth100/1/45  Down        None     Eth1/19
      Eth100/1/46  Down        None     Eth1/19
      Eth100/1/47  Down        None     Eth1/19
      Eth100/1/48  Down        None     Eth1/19
Logs:

Okay, so Eth1/19 is added back, but it’s still not associated for the fabric port for Eth100/1/25-48. Let’s redistribute manually:

N5K(config-if)# fex pinning redistribute 100
Redistributing all ports on FEX 100. This command will cause traffic disruption.
N5K(config-if)# sh fex 100 detail
FEX: 100 Description: POD16 FEX 100   state: Online
  FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
  FEX Interim version: 4.0(1a)N2(1a)
  Switch Interim version: 4.0(1a)N2(1a)
  Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
  Part No: 73-12009-05
  Card Id: 70, Mac Addr: 00:11:22:33:44:55, Num Macs: 64
  Module Sw Gen: 19 [Switch Sw Gen: 19]
 pinning-mode: static    Max-links: 2
  Fabric port for control traffic: Eth1/20
  Fabric interface state:
    Eth1/19 - Interface Down. State: Configured
    Eth1/20 - Interface Up. State: Active
  Fex Port        State  Fabric Port  Primary Fabric
       Eth100/1/1  Down     Eth1/19     Eth1/19
       Eth100/1/2  Down     Eth1/19     Eth1/19
       Eth100/1/3  Down     Eth1/19     Eth1/19
       Eth100/1/4  Down     Eth1/19     Eth1/19
       Eth100/1/5  Down     Eth1/19     Eth1/19
       Eth100/1/6  Down     Eth1/19     Eth1/19
       Eth100/1/7  Down     Eth1/19     Eth1/19
       Eth100/1/8  Down     Eth1/19     Eth1/19
       Eth100/1/9  Down     Eth1/19     Eth1/19
      Eth100/1/10  Down     Eth1/19     Eth1/19
      Eth100/1/11  Down     Eth1/19     Eth1/19
      Eth100/1/12  Down     Eth1/19     Eth1/19
      Eth100/1/13  Down     Eth1/19     Eth1/19
      Eth100/1/14  Down     Eth1/19     Eth1/19
      Eth100/1/15  Down     Eth1/19     Eth1/19
      Eth100/1/16  Down     Eth1/19     Eth1/19
      Eth100/1/17  Down     Eth1/19     Eth1/19
      Eth100/1/18  Down     Eth1/19     Eth1/19
      Eth100/1/19  Down     Eth1/19     Eth1/19
      Eth100/1/20  Down     Eth1/19     Eth1/19
      Eth100/1/21  Down     Eth1/19     Eth1/19
      Eth100/1/22  Down     Eth1/19     Eth1/19
      Eth100/1/23  Down     Eth1/19     Eth1/19
      Eth100/1/24  Down     Eth1/19     Eth1/19
      Eth100/1/25  Down     Eth1/20     Eth1/20
      Eth100/1/26  Down     Eth1/20     Eth1/20
      Eth100/1/27  Down     Eth1/20     Eth1/20
      Eth100/1/28  Down     Eth1/20     Eth1/20
      Eth100/1/29  Down     Eth1/20     Eth1/20
      Eth100/1/30  Down     Eth1/20     Eth1/20
      Eth100/1/31  Down     Eth1/20     Eth1/20
      Eth100/1/32  Down     Eth1/20     Eth1/20
      Eth100/1/33  Down     Eth1/20     Eth1/20
      Eth100/1/34  Down     Eth1/20     Eth1/20
      Eth100/1/35  Down     Eth1/20     Eth1/20
      Eth100/1/36  Down     Eth1/20     Eth1/20
      Eth100/1/37  Down     Eth1/20     Eth1/20
      Eth100/1/38  Down     Eth1/20     Eth1/20
      Eth100/1/39  Down     Eth1/20     Eth1/20
      Eth100/1/40  Down     Eth1/20     Eth1/20
      Eth100/1/41  Down     Eth1/20     Eth1/20
      Eth100/1/42  Down     Eth1/20     Eth1/20
      Eth100/1/43  Down     Eth1/20     Eth1/20
      Eth100/1/44  Down     Eth1/20     Eth1/20
      Eth100/1/45  Down     Eth1/20     Eth1/20
      Eth100/1/46  Down     Eth1/20     Eth1/20
      Eth100/1/47  Down     Eth1/20     Eth1/20
      Eth100/1/48  Down     Eth1/20     Eth1/20
Logs:

Everything’s back to normal! Well, just keep in mind that static pinning is sticky, which is why I prefer to use EtherChannels than static pinning.

Related posts:

  1. The FEX information is cached until it’s re-assigned to a different FEX Continuing our journey into the world of FEXs on the...
  2. Methods for connecting FEXs If you’re wondering what an FEX is, check out this...
  3. What happens when the wrong serial number is entered for the FEX? Continuing on the journey of FEXs on the Nexus platform,...
  4. Why EtherChannels should be used for FEX interfaces I prefer to use port-channel interfaces for the fabric interfaces...
  5. FEX configs are retained The N5k maintains the config of the FEXs, even when...

Related posts brought to you by Yet Another Related Posts Plugin.

Tags: , , , , , , , , , , , , , , , , , , , , ,

Leave a Reply

You must be logged in to post a comment.