Experimenting with Static Pinning

What happens when more links are associated with an FEX than are permitted in the max-links parameter? Let’s find out…

Let’s go back to FEX 100 (in our example environment):

This FEX is configured to allow 1 link to be pinned. What happens if we associate multiple (more than one) interfaces to this FEX?

Hmm – this obviously worked (brought up the 2148T, shows both interfaces as active). Let’s confirm using some nifty show commands:

I see a problem – there are no interfaces associated with either of the physical interfaces. Let’s try removing one of the links from the FEX and see if this resolves itself. We’ve dis-associated Eth1/20 from FEX 100:

Note that Eth1/20 is listed as Discovered – this is because it sees the N2148T (the mode is still fex-fabric for Eth1/20), so it’s pretty much indicating that it sees something there, but it’s not Active, so it’s not actively participating in the FEX.

Let’s dig deeper:

Okay, now things are coming together. Even the Po100 interface isn’t up, the FEX sees this as the one interface and ties everything to it. Let’s validate this in the config:

Now let’s remove it:

Now verify:

Bingo! Let’s check out some of the other commands we were looking at:

Now what happens if we exceed the number of links in the FEX? Let’s find out and associate Eth1/20 back with FEX 100. Here’s the output after we’ve associated Eth1/20 with FEX 100:

This is good – they’re both added to the FEX, but no ports are assigned to Eth1/20 (the port that exceeds the pinning max-links parameter). Let’s play around a little and see if we can re-distribute the interfaces (maybe we can trick NX-OS into spreading them across both, even though we only want one?):

No tricking it today!

How about if we shutdown the first interface (Eth1/19) – will it down the FEX or will it automatically spread the interfaces to the remaining interface (Eth1/20)?

No – no-go. NX-OS uses the first interface defined (whether or not it’s up). It’s statically pinned (hence the name) to the group of ports (which is determined by the pinning max-links parameter).

I have a question – is the ordering based on configuration order or does it re-order them based on the interface name (ie. Port-Channel versus Ethernet)?

Let’s find out – let’s add back in the Po100 interface:

Interesting – let’s bring back up Eth1/19 now and see what happens:

Nothing. So, the ordering of the interfaces is based on the name. Port-Channel interfaces take precedence over Ethernet interfaces, for some reason – maybe they’re sorted alphabetically in reverse? As can be seen above, there are no interfaces tied to Eth1/19 (and definitely won’t be on Eth1/20). Let’s confirm:

So because the first interface is down, NX-OS doesn’t use any remaining interfaces (because the pinning takes the first interface shown in the list of Fabric interfaces, not the first available (up/up) interface in the list). This behaves more like a static list, in that there’s no searching down the list.

Let’s adjust the max-links and see what happens:

Oh – I forgot about this! Port-Channels can only be used with a max-link of 1. We’ll now remove the Po100 interface then try it again:

Eth1/19 is back up-and-running. Now shutdown Eth1/19 and adjust the max-links parameter:

This proves what is said in the Nexus 2000 NX-OS config guide that if a fex interface is down, the associated interfaces on the 2148T (fabric extender) will remain down until the interface is resolved (restored to service). See http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_4_1/Cisco_Nexus_2000_Series_Fabric_Extender_Software_Configuration_Guide_chapter2.html#con_1203877 for more info (look at the note).

Okay, well let’s go ahead and bring up Eth1/19 now:

The note in the config guide is why NX-OS doesn’t failover between remaining fabric interfaces (because the interfaces are pinned to each interface in the order shown in the output of the show fex command).

One last item to mention here is that if two links are specified and only one link exists, only half of the FEX interfaces will work!

Eth1/19 and Eth1/20 are used in the following example. Eth1/19 has been disassociated from FEX 100, and here’s the output:

Notice how Eth100/1-24 don’t have any primary fabric interface!

Let’s try redistributing the interfaces:

Now let’s look at the details of FEX 100:

Hmm – it flip-flopped things around, so that Eth100/1/1-24 are tied to Eth1/20, but Eth100/1/25-48 are not associated with any physical interface. Bummer. We could change the max-links to 1, then all of the FEX interfaces (Eth100/1/1-48) would be tied to Eth1/20, or we could re-associate Eth1/19. This is what we’ll do – add Eth1/19 back to the FEX:

Okay, so Eth1/19 is added back, but it’s still not associated for the fabric port for Eth100/1/25-48. Let’s redistribute manually:

Everything’s back to normal! Well, just keep in mind that static pinning is sticky, which is why I prefer to use EtherChannels than static pinning.

One comment on “Experimenting with Static Pinning

  1. jatin jesse on said:

    Awesome!!!
    I learnt a lot from this detailed article.
    Keep Posting,,,, Thanks!

Leave a Reply

Your email address will not be published.

HTML tags are not allowed.

WordPress Themes