NX2000 port connectivity

Rosa Ladeira · ‎11-25-2011

I have nx2000 boxes connected to nx7000 fiber module.

There is a intermittent error occuring on nx7000 fiber module.

7000 fiber module drops down and all NX2000 boxes lose connectivity.

When 7000 fiber module comes up, some equipment, not all, can not be reached through console port via network.

These equipment are datacenter air conditioner console port, storage controller console port (iscsi controller interfaces run fine).

It seems that handshake between these equipments and NX2000 ports are not fine.

Any idea ?

Thanks, Rosa

Amit Singh · ‎11-25-2011

Hi Rosa,

Do you have any errors in the logs or console? Could you please pate that.

What is the NX-OS version you are running on the Nexus7000? What are the Nexus 2K types?

Cheers,

-amit singh

Rosa Ladeira · ‎11-25-2011

Logs

Unfortunately, when 7000 fiber module drops down internal backbone drpos down too. No messages on servers.

Immediately after connectivity comes up storage controler signs it is up. No errors.

Version

nx7000-wc0# sh version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie

s_home.html

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

Software

BIOS: version 3.22.0

kickstart: version 5.1(1)

system: version 5.1(1)

BIOS compile time: 02/20/10

kickstart image file is: bootflash:///n7000-s1-kickstart-npe.5.1.1.bin

kickstart compile time: 12/25/2020 12:00:00 [10/24/2010 08:25:25]

system image file is: bootflash:///n7000-s1-dk9-npe.5.1.1.bin

system compile time: 9/2/2010 19:00:00 [10/24/2010 09:34:48]

Hardware

cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")

Intel(R) Xeon(R) CPU with 4109596 kB of memory.

Processor Board ID JAF1444BLKS

Device name: nx7000-wc0

bootflash: 2029608 kB

slot0: 0 kB (expansion flash)

Kernel uptime is 20 day(s), 6 hour(s), 35 minute(s), 18 second(s)

Last reset at 254133 usecs after Sat Nov 5 09:47:59 2011

Reason: Reset Requested by CLI command reload

System version: 5.1(1)

Service:

plugin

Core Plugin, Ethernet Plugin

CMP (Module 5) ok

CMP Software

CMP BIOS version: 02.01.05

CMP Image version: 5.1(1) [build 5.0(0.66)]

CMP BIOS compile time: 8/ 4/2008 19:39:40

CMP Image compile time: 9/2/2010 19:00:00

nx7000-wc0#

Oleksandr Nesterov · ‎11-26-2011

Rosa, can you, please, attach:

"show logging log", "sh mod", "sh interface ethernet x/y transceiver details" from interfaces to which fex are connected

Rosa Ladeira · ‎11-28-2011

nx7000-wc0# sh mod

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 32 10 Gbps Ethernet Module N7K-M132XP-12 ok

5 0 Supervisor module-1X N7K-SUP1 active *

10 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok

Mod Sw Hw

--- -------------- ------

1 5.1(1) 2.0

5 5.1(1) 1.8

10 5.1(1) 1.6

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 c4-71-fe-a5-bb-1c to c4-71-fe-a5-bb-40 JAF1445AGRQ

5 b4-14-89-e3-6f-00 to b4-14-89-e3-6f-08 JAF1444BLKS

10 c4-71-fe-1b-d1-f4 to c4-71-fe-1b-d2-28 JAF1444CGPG

Mod Online Diag Status

--- ------------------

1 Pass

5 Pass

10 Pass

Xbar Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Fabric Module 1 N7K-C7010-FAB-1 ok

2 0 Fabric Module 1 N7K-C7010-FAB-1 ok

3 0 Fabric Module 1 N7K-C7010-FAB-1 ok

Xbar Sw Hw

--- -------------- ------

1 NA 1.1

2 NA 1.1

3 NA 1.1

Xbar MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 NA JAF1443BRQR

2 NA JAF1443BSGN

3 NA JAF1443BMPE

* this terminal session

nx7000-wc0#

Rosa Ladeira · ‎11-28-2011

nx7000-wc0# sh interface Ethernet1/10 transceiver details

Ethernet1/10

transceiver is present

type is 10Gbase-SR

name is CISCO-SUMITOMO

part number is SPP5100SR-C1

revision is A

serial number is SPC144604SH

nominal bitrate is 10300 MBit/sec

Link length supported for 50/125um OM2 fiber is 82 m

Link length supported for 50/125um OM3 fiber is 300 m

Link length supported for 62.5/125um fiber is 26 m

cisco id is --

cisco extended id number is 4

SFP Detail Diagnostics Information (internal calibration)

----------------------------------------------------------------------------

Current Alarms Warnings

Measurement High Low High Low

----------------------------------------------------------------------------

Temperature 26.74 C 75.00 C -5.00 C 70.00 C 0.00 C

Voltage 3.32 V 3.63 V 2.97 V 3.46 V 3.13 V

Current 5.30 mA 10.50 mA 2.00 mA 9.00 mA 2.50 mA

Tx Power -2.65 dBm 1.69 dBm -11.30 dBm -1.30 dBm -7.30 dBm

Rx Power -5.00 dBm 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm

Transmit Fault Count = 1

----------------------------------------------------------------------------

Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

nx7000-wc0#

Rosa Ladeira · ‎11-28-2011

Ethernet111/1/1 -> storage controller console port

2011 Nov 23 11:01:45 -> intermittent error occuring on nx7000 fiber module (storage administrative port looses its IP)

2011 Nov 23 11:04:49 -> fiber module comming on

2011 Nov 25 10:46:40 -> Forcing storage IP via external host as console on storage port

nx7000-wc0# sh logging log | grep "Nov 2" | grep "Interface Ethernet111/1/1"

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/1 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/10 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/13 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/16 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/17 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/18 is down (module removed)

2011 Nov 23 11:01:45 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/19 is down (module removed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/11 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/12 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/14 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/15 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/1 is down (Interface remove

d)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/10 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/13 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/16 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/17 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/18 is down (Interface remov

ed)

2011 Nov 23 11:01:46 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/19 is down (Interface removed)

--------------------> fiber module comming on

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/13, operational speed changed to 100 Mbps

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/13, operational duplex mode changed to Full

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/13, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/13, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/13 is up in mode access

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/16, operational speed changed to 100 Mbps

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/16, operational duplex mode changed to Full

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/16, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/16, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:04:49 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/16 is up in mode access

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/10, operational speed changed to 1 Gbps

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/10, operational duplex mode changed to Full

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/10, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/10, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/10 is up in mode access

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/18, operational speed changed to 1 Gbps

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/18, operational duplex mode changed to Full

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/18, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/18, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/18 is up in mode access

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/19, operational speed changed to 1 Gbps

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/19, operational duplex mode changed to Full

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/19, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/19, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:04:50 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/19 is up in mode access

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/10 is down (module removed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/13 is down (module removed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/16 is down (module removed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/18 is down (module removed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet111/1/19 is down (module removed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/1 is down (Interface remove

d)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/11 is down (Interface remov

ed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/12 is down (Interface remov

ed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/14 is down (Interface remov

ed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/15 is down (Interface remov

ed)

2011 Nov 23 11:05:26 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/17 is down (Interface remov

ed)

2011 Nov 23 11:05:27 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/10 is down (Interface remov

ed)

2011 Nov 23 11:05:27 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/13 is down (Interface remov

ed)

2011 Nov 23 11:05:27 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/16 is down (Interface remov

ed)

2011 Nov 23 11:05:27 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/18 is down (Interface remov

ed)

2011 Nov 23 11:05:27 nx7000-wc0 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet111/1/19 is down (Interface remov

ed)

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/13, operational speed changed to 100 Mbps

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/13, operational duplex mode changed to Full

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/13, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/13, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/13 is up in mode access

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/16, operational speed changed to 100 Mbps

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/16, operational duplex mode changed to Full

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/16, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/16, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:08:24 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/16 is up in mode access

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/10, operational speed changed to 1 Gbps

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/10, operational duplex mode changed to Full

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/10, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/10, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/10 is up in mode access

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/18, operational speed changed to 1 Gbps

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/18, operational duplex mode changed to Full

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/18, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/18, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/19, operational speed changed to 1 Gbps

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/19, operational duplex mode changed to Full

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/19, operational Receive Flow Contr

ol state changed to off

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/19, operational Transmit Flow Cont

rol state changed to on

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/18 is up in mode access

2011 Nov 23 11:08:25 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/19 is up in mode access

2011 Nov 25 10:46:35 nx7000-wc0 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet111/1/17 is down (Administratively down)

Forcing storage IP via external host as console on storage port

2011 Nov 25 10:46:40 nx7000-wc0 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet111/1/17 is admin up .

2011 Nov 25 14:26:06 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/1, operational speed changed to 100 Mbps

2011 Nov 25 14:26:06 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/1, operational duplex mode changed to Full

2011 Nov 25 14:26:06 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/1, operational Receive Flow Contro

l state changed to off

2011 Nov 25 14:26:06 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/1, operational Transmit Flow Contr

ol state changed to on

2011 Nov 25 14:26:06 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/1 is up in mode access

2011 Nov 25 14:26:23 nx7000-wc0 %ETHPORT-5-SPEED: Interface Ethernet111/1/17, operational speed changed to 100 Mbps

2011 Nov 25 14:26:23 nx7000-wc0 %ETHPORT-5-IF_DUPLEX: Interface Ethernet111/1/17, operational duplex mode changed to Full

2011 Nov 25 14:26:23 nx7000-wc0 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet111/1/17, operational Receive Flow Contr

ol state changed to off

2011 Nov 25 14:26:23 nx7000-wc0 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet111/1/17, operational Transmit Flow Cont

rol state changed to on

2011 Nov 25 14:26:23 nx7000-wc0 %ETHPORT-5-IF_UP: Interface Ethernet111/1/17 is up in mode access

nx7000-wc0#

Rosa Ladeira · ‎12-01-2011

air conditioner was attached after nx7000 fiber module crashes.