FEX configs are retained

The N5k maintains the config of the FEXs, even when they’re removed (and even though it’s not visible to us). In this article, we’re going to look into this a little further… We’ll start with a working FEX, using Po100 (with Eth1/19-20 being the physical bundle members).

We can see the config for eth100/1/1:

N5K(config-if)# sh run int eth 100/1/1
version 4.0(1a)N2(1a)

interface Ethernet100/1/1
  switchport access vlan 1016

N5K(config-if)#

Now let’s shut down the Port-Channel:

N5K(config-if)# int eth 1/19-20
N5K(config-if-range)# shut
2010 Jan 27 17:01:02 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL)
N5K(config-if-range)#

Now look for the eth100/1/1 config:

N5K(config-if-range)# sh run int eth 100/1/1
                                      ^
% invalid command detected at '^' marker.
Invalid range
N5K(config-if-range)#

Oh no, we’re not in Kansas anymore!!! Don’t worry – it’s there. Let’s bring up the FEX now and see it come back, automagically:

N5K(config-if-range)# no shut
N5K(config-if-range)# 2010 Jan 27 17:01:42 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL)

N5K(config-if-range)# sh run int eth 100/1/1
version 4.0(1a)N2(1a)

interface Ethernet100/1/1
  switchport access vlan 1016

N5K(config-if-range)#

Ah, life’s great! We’re back in business.

I was curious about this, if there’s a private config file or something that’s used to maintain these invisible configurations. Poking around, I looked at:

N5K(config-if-range)# dir
         48     Jul 13 02:11:43 2009  ..tmp-kickstart
         38     Jul 13 02:11:43 2009  ..tmp-system
   21524480     Jul 13 02:11:43 2009  .tmp-kickstart
  103586147     Jul 13 02:11:43 2009  .tmp-system
      49152     Jul 13 02:11:43 2009  lost+found/
   21524480     Jul 13 02:11:03 2009  n5000-uk9-kickstart.4.0.1a.N2.1a.bin
  103586147     Jul 13 02:08:42 2009  n5000-uk9.4.0.1a.N2.1a.bin
       4096     Jan 01 00:03:16 2005  routing-sw/

Usage for bootflash://
  312094720 bytes used
  568389632 bytes free
  880484352 bytes total
N5K(config-if-range)# dir bootflash:/routing-sw/
       4096     Jan 01 00:03:16 2005  config/

Usage for bootflash://sup-local
  312094720 bytes used
  568389632 bytes free
  880484352 bytes total
N5K(config-if-range)# dir bootflash:/routing-sw/config/

Usage for bootflash://sup-local
  312094720 bytes used
  568389632 bytes free
  880484352 bytes total
N5K(config-if-range)#

It could be that the config is a hidden/system file that I can’t see, although I don’t see any flags for the dir command. Anyway, it’s obvious that it keeps the config, just that it’s hidden until the fabric extender comes back online.

There is a caveat – if the FEX is removed, the config is blown away (lost):

N5K(config-if-range)# no int po100
2010 Jan 27 17:08:32 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL)
N5K(config)#
N5K(config)# int e1/19-20
N5K(config-if-range)# no switch mode
N5K(config-if-range)# no fex assoc 100
N5K(config-if-range)# no fex 100
N5K(config)# fex 100
N5K(config-fex)# pin max 2
Change in Max-links will cause traffic disruption.
N5K(config-fex)# desc "POD16 FEX 100"
N5K(config-fex)# int eth1/19-20
N5K(config-if-range)# switch mode fex
N5K(config-if-range)# fex associ 100
N5K(config-if-range)# vlan 1016
N5K(config-vlan)# int2010 Jan 27 17:09:22 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL)
 eth100/1/1
N5K(config-if)# exit
N5K(config)# sh run int eth100/1/1
version 4.0(1a)N2(1a)

interface Ethernet100/1/1

N5K(config)#

The moral of the story is to not remove the FEX from the Nexus 5000 if you don’t want the config of the fabric extenders (FEXs – 2148Ts at this point) to be lost. I haven’t had a chance to experiment with swapping out 2148Ts, to see if a change in the serial number would make a difference. I don’t think it would (as long as the s/n isn’t specified so that a different FEX can’t peer with the parent switch) – I might try that scenario another day.