Continuing our journey into the world of FEXs on the Nexus 5000 and 2000, today we’re going to look at the behavior of the FEX configs themselves on the Nexus 5000.
All of the configuration and software information (firmware/images) for the FEX (Nexus 2148T) are kept on the parent switch (a Nexus 5000-series switch). Going on this, what happens when the FEX goes offline? Do I lose my config?
A stack of 3750 switches can be provisioned ahead of time, so that as stack members are added (assuming that they’re the correct type/model), no changes must be made – just move on. What kind of behavior is available on the Nexus 5000/2000?
I haven’t found provisioning available yet for FEXs on the N5k, however I did notice that the configurations for the FEX are retained when the FEX goes offline and online. This is great — if the 2148T fails or the fabric interface links are disconnected, the config will still be there (although not visible until the FEX comes back online). This article is looking at it from an association perspective (N2148T association with the N5k, not a configuration (actual port configs on the FEX). With that said, let’s explore this further…
Using the same example we’ve been using in this series (the EtherChannel fabric interface example), we’re going to take down the fabric interface:
N5K(config)# int ether 1/19-20 N5K(config-if-range)# no channel-group 2010 Jan 27 14:56:57 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL)
Good! We’re using an EtherChannel, so this should take down the fabric interface and allow us to continue with the FEX offline. Let’s verify that the FEX is offline:
N5K(config-if-range)# show fex 100
FEX: 100 Description: First_Nexus_2148T state: Offline
FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
Part No: 73-12009-05
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/20
Fabric interface state:
Po100 - Interface Down. State: Configured
Eth1/19 - Interface Up. State: Active
Eth1/20 - Interface Up. State: Active
N5K(config-if-range)#
Great! We’re good to go…
N5K(config-if-range)# 2010 Jan 27 14:57:35 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number PRIVATE_SERIAL)
What?! I removed the port-channel from the physical interfaces (effectively taking the port-channel down), what’s going on?! Why is FEX 100 online now? Maybe this was a bug – let’s look at it in more detail:
N5K(config-if-range)# show fex 100
FEX: 100 Description: First_Nexus_2148T state: Online
FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
Part No: 73-12009-05
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/20
Fabric interface state:
Po100 - Interface Down. State: Configured
Eth1/19 - Interface Up. State: Active
Eth1/20 - Interface Up. State: Active
Oh – the physical interfaces are coming up (not using the EtherChannel). Let’s look at the running-config and see what’s on these interfaces:
N5K(config-if-range)# show run int ether 1/19 version 4.0(1a)N2(1a) interface Ethernet1/19 switchport mode fex-fabric fex associate 100 N5K(config-if-range)# show run int ether 1/20 version 4.0(1a)N2(1a) interface Ethernet1/20 switchport mode fex-fabric fex associate 100
Sure enough, because we were using an EtherChannel (which requires a pinning max-links value of 1), the switch is taking the first interface (Eth1/19), noticing that it’s up and bringing the FEX up on the physical interface. We need to remove the FEX association from ether 1/19-20:
N5K(config-if-range)# no fex associate 100
N5K(config-if-range)# 2010 Jan 27 14:58:27 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number PRIVATE_SERIAL)
N5K(config-if-range)# show fex 100
FEX: 100 Description: First_Nexus_2148T state: Offline
FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
Part No: 73-12009-05
pinning-mode: static Max-links: 1
Fabric port for control traffic:
Fabric interface state:
Po100 - Interface Down. State: Configured
N5K(config-if-range)#
Now it’s down for good – note that the FEX 100 info is still cached (model, serial number, etc.)?
Now let’s assign this same 2148T to a different FEX (101) and see if the cached state remains:
N5K(config)# int ether 1/19-20 N5K(config-if-range)# switch mode fex N5K(config-if-range)# fex associate 101 N5K(config-if-range)# 2010 Jan 27 15:02:59 N5K %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-line (Serial Number PRIVATE_SERIAL)
FEX 100 is gone (the cached info), but FEX 101 is showing up as active:
N5K(config-if-range)# show fex 100
FEX 100 not found
N5K(config-if-range)# show fex 101
FEX: 101 Description: Non_PC_Fex_Config state: Online
FEX version: 4.0(1a)N2(1a) [Switch version: 4.0(1a)N2(1a)]
Extender Model: N2K-C2148T-1GE, Extender Serial: PRIVATE_SERIAL
Part No: 73-12009-05
pinning-mode: static Max-links: 2
Fabric port for control traffic: Eth1/19
Fabric interface state:
Eth1/19 - Interface Up. State: Active
Eth1/20 - Interface Up. State: Active
N5K(config-if-range)#
I’m guessing that if there were multiple 2148Ts, and one were to go down, as long as the same unit (same serial number) is not re-connected to a different FEX ID, the cached information would remain? I’m not sure how NX-OS indexes the FEXs, but it seems to me that it’s more serial-number-based.
As I experiment with this further, I’ll know for sure, but this is my hypothesis… ’til next time.
Related posts:
- What happens when the wrong serial number is entered for the FEX? Continuing on the journey of FEXs on the Nexus platform,...
- Methods for connecting FEXs If you’re wondering what an FEX is, check out this...
- Experimenting with Static Pinning What happens when more links are associated with an FEX...
- FEX configs are retained The N5k maintains the config of the FEXs, even when...
- Why EtherChannels should be used for FEX interfaces I prefer to use port-channel interfaces for the fabric interfaces...
Related posts brought to you by Yet Another Related Posts Plugin.
Tags: 2000, 2148T, 5000, 5010, 5020, associate, cisco, config, configuration, etherchannel, ethernet, example, fabric extender, fabric interface, fex, max-links, nexus, nexus 2000, nexus 2148T, nexus 5000, nexus 5010, nexus 5020, nx-os, pinning, training, tutorial