cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5106
Views
5
Helpful
12
Replies

Server VIF Paths FEX Host & Network Port Pinning

b.e.leblanc
Level 1
Level 1

Hello,

I have a UCS-B Series Chassis with 4 B200 M2 Blades connecting to 2 6124 FI's with 4 Server Ports to each FI.

I'm wondering if anyone can help me understand VIF Paths & FEX "Pinning"?

Here's a screenshot of the VIF Paths tab for Server 1:

VIF Paths.JPG

From my understanding each blade has two FEX Host Ports (right & left) and those pin to FEX Network Ports (aka Server Ports).  Unlike SAN & LAN pinning I can't seem to figure out how to specify FEX Port pinning or control its behavior.

The reason why I'm wondering this is because I was doing a Test Plan Execution and pulled the right/1 & left/1 FEX Network Port silmultaniously.  This caused me to lose connectivity to my Server 1 that was pinned to those ports.  I would have thought the blade would "automatically" fail-to/use the next available FEX Network Port, either right/2, right/3, right/4 or left/2, left/3, left/4 but this wasn't the case.

Can anyone please explain?

Thanks in advance,

-Brian

1 Accepted Solution

Accepted Solutions

lwatta
Cisco Employee
Cisco Employee

Pinning from the FEX/IOM to the Blades is hard coded by the FEX during initialization. It's not user configurable. The pinning is determined at initialization depending on the number links between the FEX and the FI.

Now when you pull a link between a FEX and FI we never ever repin the links on failure. So I'm not surprised that you lost network connectivty when you pulled all the links. If you re-ack the chassis after you a pull a FEX link it will repin the connections but we never do it automatically. Keep in mind a re-ack will mean the blades will take a short network outage while the connections are repinned.

With the second generation FI (6248) and second generation IOM (2208) you can create a port-channel between the FI and IOM. In this case if one link fails the traffic will get re-balanced across the other links.

So to recap.

Gen 1 hardware everything is pinned automatically by the IOM. Failure of a link means those servers lose network connectivity on that fabric. A chassis re-ack will fix the problem but could cause a short network outage on the other servers

Gen 2 hardware has the capability to do a port-channel between the FI and IOM. In this case pulling a cable between FI and IOM should not cause a network outage of the server.

Hope that helps.

louis

View solution in original post

12 Replies 12

lwatta
Cisco Employee
Cisco Employee

Pinning from the FEX/IOM to the Blades is hard coded by the FEX during initialization. It's not user configurable. The pinning is determined at initialization depending on the number links between the FEX and the FI.

Now when you pull a link between a FEX and FI we never ever repin the links on failure. So I'm not surprised that you lost network connectivty when you pulled all the links. If you re-ack the chassis after you a pull a FEX link it will repin the connections but we never do it automatically. Keep in mind a re-ack will mean the blades will take a short network outage while the connections are repinned.

With the second generation FI (6248) and second generation IOM (2208) you can create a port-channel between the FI and IOM. In this case if one link fails the traffic will get re-balanced across the other links.

So to recap.

Gen 1 hardware everything is pinned automatically by the IOM. Failure of a link means those servers lose network connectivity on that fabric. A chassis re-ack will fix the problem but could cause a short network outage on the other servers

Gen 2 hardware has the capability to do a port-channel between the FI and IOM. In this case pulling a cable between FI and IOM should not cause a network outage of the server.

Hope that helps.

louis

bassettkyle
Level 1
Level 1

While back I also found gui report bugs in vif paths tab in 2.01q, confirmed with tac. Addressed in t.

Updated and it was fixed.

b.e.leblanc
Level 1
Level 1

Louis,

Thanks for your great response.  I makes perfect sense and so does using a Port Channel in v2.0 of the Hardware.

I have 2 other follow up questions:

1. If the cables are reconnected will the blades/chassis start using them or will you have to re-ack again?

2. Part of the test plan I was executing had a scenario where an administrator reconnected the cables but in reverse.  In other words you would the original 3 going from FEX-a to FI-a & FEX-b to FI-b but 1 cable from each going from FEX-a to FI-b & FEX-b to FI-a.  When the chassis would be re-ack would that cause huge problems?

Here's a visual:

Thanks,

-Brian

Brian, in your diagram you have a link from FEX-A to FI-B and from FEX-B to FI-B. This is an unsupported configuration, FEX-A can only be connected to FI-A and FEX-B can only be connected to FI-B. Also having only 3 fabric ports connected is unsupported, the supported configuration is 1, 2 or 4. In 95% of the installs we do only 2 fabric ports are used as having 4 cuts the supported chassis scalability in half. We have yet to see any oversubscription issues with having only 2 fabric ports connected.

Hey Jeremy,

The diagram above is meant as a "scenario" where an administrator mistakenly plugged the cables in the wrong ports (i.e. reversed them).  In executing a test plan prior to going live we want to understand expected behavior in the rare case someone has mis-cable by accident.

Why does having 4 fabric ports cut the supported chassis scalability in half?  My intent in having cabled all 4 initially is to not have to add any cables in the future when we add more blades.  A true no cable upgrade, if you catch my drift.

In the event of a failure on just 1 of the fabric ports and after re-ack having the chassis run in the 3 port "unsupported" configuration everything will still continue to work until the remaining single fabric ports are "fixed" or it is best practice to pull the second fabric ports and re-ack the chassis to be in a supported configuration 1 or 2 fabric ports?

-Brian

It cuts the supported chassis count in half because you are using twice the FI ports. If you have a 6120 and cable up all 4 fabric ports you could only ever have a max of 5 chassis. When using 2 fabric cables your chassis count is 10. It also requires more FI port licenses. In most environments 4 fabric connections is way over kill when 2 provide plenty of bandwidth and then some.

Jeremy,

Great points!

I will consider using only 2 in the future for sure.

-Brian

Funny this topic comes up because we were just talking about it internally which caused me to test a few of these scenarios.

For #1 When you plug the cable back in it starts using it immediately. You do not have to re-ack.

For #2 That is an unsupported configuration and I beleive the FI will reject the connection. I will test to be sure and let you know.

louis

lwatta
Cisco Employee
Cisco Employee

I just tested #2. It FI doesn't like that scenario at all. It knows that the cable is connected into the wrong FEX and will not allow the port to be usable.

Whats interesting is it will not even let you re-ack the chassis because it knows the connections are wrong.

Louis,

Interesting that it automatically re-uses upon re-connection!  So, am I to understand it will re-balance the FEX to Blade pinning?

Strange it does it automatically upon re-connection but not automatically upon disconnection.  In all honesty it would make more sense to have to manually re-ack upon connection as most likely someone is "choosing" to do that, where upon failure that would normally happen unattended, if you know what I mean.

Idealy I would hope the FI's would reject the connection to avoid any possible problems or confusion.

Thanks for offering to test and I look forward to the results.

-Brian

When a cable is pulled the FEX does not re-pinn. If fabric failover is enabled and the mezzanine card supports it, we will try to push the packets up the other FEX. So there is no re-pinn of the FEX links when a connection is lost.

The reason we don't re-pinn is that it could cause a network disruption for the other servers during that re-pinn.

Now if you manually run a re-ack while a connection is failed we will re-pinn all the connections. This will cause a short network outage for some of the servers but it will enable the servers that had a failed connection to be able to start using that fabric again. The bandwidth will be decreased but all the servers will have connectivity to both fabrics.

The good news is that the ability to Port-channel with the new hardware makes this subject so much easier to understand :-)

louis

Louis,

I can't thank you enough for your help & explanations.

-Brian

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: