We replaced the fex twice and

t844311Imran · ‎07-21-2015

I have one nexus 7000 with F2 blade and a new FEX 2248TP. I have setup all FEX configuration but the communication between 7000 and 2000 is down. The physical connection doesn't work.

We do have the SFP-10GSR available and we already tested on that. I see the interfaces flapping. The issue is only occurring on one new FEX only.

Core# sh int e4/29-30 | i flapped
Last link flapped 00:00:51
Last link flapped 00:00:05
Core# sh int e4/29-30 | i flapped
Last link flapped 00:00:01
Last link flapped 00:00:07

Ethernet4/29 is down (initializing)
Ethernet4/30 is down (Link not connected)
Ethernet3/29 is down (Link not connected)
Ethernet3/30 is down (initializing)

2015 Jul 21 10:38:01.104 Core %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet3/29 is down (Initializing)
2015 Jul 21 10:38:01.107 Core %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 199 that is connected with Ethernet3/29 ch
anged its status from Active to Disconnected
2015 Jul 21 10:38:01.272 Core %ETHPORT-5-SPEED: Interface Ethernet4/29, operational speed changed to 10 Gbps
2015 Jul 21 10:38:01.272 Core %ETHPORT-5-IF_DUPLEX: Interface Ethernet4/29, operational duplex mode changed to Full
2015 Jul 21 10:38:01.272 Core %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet4/29, operational Receive Flow Control st
ate changed to off
2015 Jul 21 10:38:01.272 Core %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet4/29, operational Transmit Flow Control s
tate changed to off
2015 Jul 21 10:38:01.294 Core %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 199 that is connected with Ethernet4/29 ch
anged its status from Configured to Fabric Up
2015 Jul 21 10:38:02.029 Core %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 199 that is connected with Ethernet4/29 ch
anged its status from Fabric Up to Connecting
2015 Jul 21 10:38:02.055 Core %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 199 that is connected with Ethernet4/29 ch
anged its status from Connecting to Active
2015 Jul 21 10:38:02.229 Core %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 199 that is connected with Ethernet3/29 ch
anged its status from Disconnected to Configured
We change the ports on same module and SFPs but the problem is the same. Can you help?

Madhukrishnan Gopinathan Nair · ‎07-22-2015

Did you bundle these interfaces together?

Can you share the configuration please.

Thanks,

Madhu

t844311Imran · ‎07-22-2015

N7K# sh run int po199 mem

!Command: show running-config interface port-channel199 membership
!Time: Wed Jul 22 15:47:30 2015

version 6.2(10)

interface port-channel199
switchport mode fex-fabric
fex associate 199
mtu 9216

interface Ethernet3/29
switchport mode fex-fabric
fex associate 199
beacon
mtu 9216
channel-group 199
no shutdown

interface Ethernet4/29
switchport mode fex-fabric
fex associate 199
beacon
mtu 9216
channel-group 199
no lacp suspend-individual
no shutdown

interface Ethernet3/30
switchport mode fex-fabric
fex associate 199
beacon
mtu 9216
channel-group 199
no shutdown

interface Ethernet4/30
switchport mode fex-fabric
fex associate 199
beacon
mtu 9216
channel-group 199
no lacp suspend-individual
no shutdown

N7K#

Madhukrishnan Gopinathan Nair · ‎07-22-2015

Do you have another fex to check, may be a faulty fex.

Thanks,

Madhu

Madhukrishnan Gopinathan Nair · ‎07-22-2015

Or can you reload this fex and check if it resolves it.

Madhu

t844311Imran · ‎07-22-2015

FEX 158 was the original FEX# which failed and after failures we tried a new FEX# 199 and port-channel# 199.

Madhukrishnan Gopinathan Nair · ‎07-23-2015

Did you try changing the uplink ports on parent?

Try to bring up the FEX with only one link in Port-channel and see what happens.

Madhu

t844311Imran · ‎07-23-2015

Yes I tried changing the pair from 3/23-24, 4/23-24 to 3/29-30, 4/29-30 and tried with one link only also but to no avail. As per TAC, "It sounds like an index pointer is stale or not updated, but it’s not clear where it is exactly. When the links have moved we see them all go down for SDP timeout reasons from the satmgr process, I want to compare the counters against what you just sent in to see if they’re incrementing or not." No response after that from TAC...intriguing indeed.

t844311Imran · ‎07-22-2015

We replaced the fex twice and verified the SFPs on both sides. We don't want to touch a working production FEX as this FEX has been down for more than 2 weeks now. Management is loosing faith in support. This is a production environment so I can't mess with working stuff. We reloaded the down FEX multiple times already.

I made one interesting observation that even though FEX 158 is nowhere in running config it still shows up in env, not sure why:

N7K# sh environment fex 158

show_env_temperature_fex:1050 mts_send failed, mts_errno=107

show_env_fan_fex:1118 mts_drop failed, mts_errno=107

Power Supply Fex 158:

---------------------------------------------------------------------------

Voltage: 12 Volts

-----------------------------------------------------

PS Model Power Power Status

(Watts) (Amp)

-----------------------------------------------------

1 -- -- -- --

2 -- -- -- --

Mod Model Power Power Power Power Status

Requested Requested Allocated Allocated

(Watts) (Amp) (Watts) (Amp)

--- ------------------- ------- ---------- --------- ---------- ----------

1 0.00 0.00 0.00 0.00 powered-up

Power Usage Summary:

--------------------

Power Supply redundancy mode: redundant

Total Power Capacity 0.00 W

Power reserved for Supervisor(s) 0.00 W

Power currently used by Modules 0.00 W

-------------

Total Power Available 0.00 W

-------------

N7K# sh environment fex 199

ERROR: Fex 199 is not present.

Nexus 2K FEX not coming up and interfaces flapping after connecting to Nexus 7K