The N5k maintains the config of the FEXs, even when they’re removed (and even though it’s not visible to us). In this article, we’re going to look into this a little further… We’ll start with a working FEX, using Po100 (with Eth1/19-20 being the physical bundle members).
We can see the config for eth100/1/1:
N5K(config-if)# sh run int eth 100/1/1 version 4.0(1a)N2(1a) interface Ethernet100/1/1 switchport access vlan 1016 N5K(config-if)#
Now let’s shut down the Port-Channel:
N5K(config-if)# int eth 1/19-20 N5K(config-if-range)# shut 2010 Jan 27 17:01:02 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL) N5K(config-if-range)#
Now look for the eth100/1/1 config:
N5K(config-if-range)# sh run int eth 100/1/1
^
% invalid command detected at '^' marker.
Invalid range
N5K(config-if-range)#
Oh no, we’re not in Kansas anymore!!! Don’t worry – it’s there. Let’s bring up the FEX now and see it come back, automagically:
N5K(config-if-range)# no shut N5K(config-if-range)# 2010 Jan 27 17:01:42 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL) N5K(config-if-range)# sh run int eth 100/1/1 version 4.0(1a)N2(1a) interface Ethernet100/1/1 switchport access vlan 1016 N5K(config-if-range)#
Ah, life’s great! We’re back in business.
I was curious about this, if there’s a private config file or something that’s used to maintain these invisible configurations. Poking around, I looked at:
N5K(config-if-range)# dir
48 Jul 13 02:11:43 2009 ..tmp-kickstart
38 Jul 13 02:11:43 2009 ..tmp-system
21524480 Jul 13 02:11:43 2009 .tmp-kickstart
103586147 Jul 13 02:11:43 2009 .tmp-system
49152 Jul 13 02:11:43 2009 lost+found/
21524480 Jul 13 02:11:03 2009 n5000-uk9-kickstart.4.0.1a.N2.1a.bin
103586147 Jul 13 02:08:42 2009 n5000-uk9.4.0.1a.N2.1a.bin
4096 Jan 01 00:03:16 2005 routing-sw/
Usage for bootflash://
312094720 bytes used
568389632 bytes free
880484352 bytes total
N5K(config-if-range)# dir bootflash:/routing-sw/
4096 Jan 01 00:03:16 2005 config/
Usage for bootflash://sup-local
312094720 bytes used
568389632 bytes free
880484352 bytes total
N5K(config-if-range)# dir bootflash:/routing-sw/config/
Usage for bootflash://sup-local
312094720 bytes used
568389632 bytes free
880484352 bytes total
N5K(config-if-range)#
It could be that the config is a hidden/system file that I can’t see, although I don’t see any flags for the dir command. Anyway, it’s obvious that it keeps the config, just that it’s hidden until the fabric extender comes back online.
There is a caveat – if the FEX is removed, the config is blown away (lost):
N5K(config-if-range)# no int po100 2010 Jan 27 17:08:32 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL) N5K(config)# N5K(config)# int e1/19-20 N5K(config-if-range)# no switch mode N5K(config-if-range)# no fex assoc 100 N5K(config-if-range)# no fex 100 N5K(config)# fex 100 N5K(config-fex)# pin max 2 Change in Max-links will cause traffic disruption. N5K(config-fex)# desc "POD16 FEX 100" N5K(config-fex)# int eth1/19-20 N5K(config-if-range)# switch mode fex N5K(config-if-range)# fex associ 100 N5K(config-if-range)# vlan 1016 N5K(config-vlan)# int2010 Jan 27 17:09:22 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL) eth100/1/1 N5K(config-if)# exit N5K(config)# sh run int eth100/1/1 version 4.0(1a)N2(1a) interface Ethernet100/1/1 N5K(config)#
The moral of the story is to not remove the FEX from the Nexus 5000 if you don’t want the config of the fabric extenders (FEXs – 2148Ts at this point) to be lost. I haven’t had a chance to experiment with swapping out 2148Ts, to see if a change in the serial number would make a difference. I don’t think it would (as long as the s/n isn’t specified so that a different FEX can’t peer with the parent switch) – I might try that scenario another day.
Related posts:
- The FEX information is cached until it’s re-assigned to a different FEX Continuing our journey into the world of FEXs on the...
- What happens when the wrong serial number is entered for the FEX? Continuing on the journey of FEXs on the Nexus platform,...
- Why EtherChannels should be used for FEX interfaces I prefer to use port-channel interfaces for the fabric interfaces...
- Methods for connecting FEXs If you’re wondering what an FEX is, check out this...
- Experimenting with Static Pinning What happens when more links are associated with an FEX...
Related posts brought to you by Yet Another Related Posts Plugin.
Tags: 2000, associate, cisco, config, configuration, example, extender, fabric, fabric extender, fex, fex associate, keep, n2148t, n2k, n5k, nexus, nexus 2000, nexus 2148T, nexus 5000, nexus 5010, nexus 5020, nx-os, retain, retained, store, training, tutorial