Several ports on 3750 are in "err-disabled" state

spaansj05 · ‎04-08-2013

We have a C3750E-24PD and a C3750G-24PS which are stacked. Both are running IOS version 15.0(2)SE2.

The 3750E is Switch 1 in the stack.

We have several ports which have gone into err-disabled state repeatedly. Both on Switch 1 and Switch 2 of the stack.

The first time it happened, I thought maybe the users had plugged a malfunctioning device into several different jacks in an attempt to get it to work but I have verified that this is not and end-user device or cabling problem.

"sh log" reveals the following.

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 6223 messages logged, xml disabled,

filtering disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 6223 messages logged, xml disabled,

filtering disabled

Exception Logging: size (4096 bytes)

Count and timestamp logging messages: disabled

File logging: disabled

Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 6217 message lines logged

Logging to 192.236.37.155 (udp port 514, audit disabled,

link up),

6217 message lines logged,

0 message lines rate-limited,

0 message lines dropped-by-MD,

xml disabled, sequence number disabled

filtering disabled

Logging Source-Interface: VRF Name:

Log Buffer (4096 bytes):

e: 0.00 V, Threshold value: 2.96 V.

Apr 8 10:43:50.599: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 10:53:50.944: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:03:51.273: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:13:51.643: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:23:52.005: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:33:52.334: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:43:52.629: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 11:53:52.916: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:03:53.303: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:13:53.708: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:23:54.020: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:33:54.340: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:43:54.652: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 12:53:54.964: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:03:55.251: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:13:55.546: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:23:55.883: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:33:56.203: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:43:56.582: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 13:53:56.978: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:03:57.374: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:13:57.720: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:23:58.006: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:33:58.343: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:43:58.605: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 14:53:59.001: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 15:03:59.354: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 15:13:59.658: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 15:24:00.020: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 15:34:00.315: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Apr 8 15:44:00.744: %SFF8472-5-THRESHOLD_VIOLATION: Te1/0/1: Voltage low alarm; Operating value: 0.00 V, Threshold value: 2.96 V.

Te1/0/1 is connected via 10Gbps to a 4900M. The 4900M is running IOS version 12.2(52)SG

Any help is appreciated.

Bilal Nawaz · ‎04-08-2013

I recommend trying a brand new SFP in both ends, as well as a new fiber cable. What is happening on the 4900M side? And am i correct in saying this is 10Gb for both sides? What is the distance between both devices?

'show int te1/0/1 transceiver detail'

'show idprom te1/0/1'

What is the output for this please?

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

spaansj05 · ‎04-08-2013

I am not seeing any problems on the 4900M though I have not dug real deep. Yes, 10Gbps on both sides. There is about a 1/4 mile between these two devices.

3750 output

CV1-166-01-3750E#sh int te1/0/1 transceiver detail

ITU Channel not available (Wavelength not available),

Transceiver is internally calibrated.

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, + : high warning, - : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm

Temperature Threshold Threshold Threshold Threshold

Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)

--------- ------------------ ---------- --------- --------- ---------

Te1/0/1 36.3 74.0 70.0 0.0 -4.0

High Alarm High Warn Low Warn Low Alarm

Voltage Threshold Threshold Threshold Threshold

Port (Volts) (Volts) (Volts) (Volts) (Volts)

--------- --------------- ---------- --------- --------- ---------

Te1/0/1 0.00 3.63 3.46 3.13 2.96

Optical High Alarm High Warn Low Warn Low Alarm

Transmit Power Threshold Threshold Threshold Threshold

Port (dBm) (dBm) (dBm) (dBm) (dBm)

--------- ----------------- ---------- --------- --------- ---------

Te1/0/1 -1.8 4.5 0.5 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm

Receive Power Threshold Threshold Threshold Threshold

Port (dBm) (dBm) (dBm) (dBm) (dBm)

------- ----------------- ---------- --------- --------- ---------

Te1/0/1 -2.7 4.5 0.5 -14.1 -18.4

CV1-166-01-3750E#sh idprom ?

interface Specify an interface for idprom display

CV1-166-01-3750E#sh idprom int te1/0/1

SEEPROM contents (hex) size 256:

0x00: 1E 01 00 01 00 0B 77 A7 01 00 00 02 01 01 28 48

0x10: 01 02 00 00 00 00 00 00 00 00 00 03 E8 20 00 01

0x20: FF B8 00 00 00 00 00 00 00 00 00 00 C0 9B E0 00

0x30: 03 E8 00 43 49 53 43 4F 2D 45 58 43 45 4C 49 47

0x40: 48 54 20 53 44 58 34 31 30 32 4C 52 2D 43 31 2D

0x50: 4D 20 20 41 20 45 43 4C 31 33 35 30 30 32 36 56

0x60: 20 20 20 20 20 32 30 30 39 31 32 31 34 35 41 00

0x70: 01 02 04 C7 01 00 18 58 32 2D 31 30 47 42 2D 4C

0x80: 52 20 20 20 20 20 20 20 56 30 34 20 45 43 4C 31

0x90: 33 35 30 30 32 36 56 31 30 2D 32 30 33 36 2D 30

0xA0: 34 20 20 41 30 20 20 00 00 00 00 00 00 00 00 00

0xB0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0xC0: 84 01 8A 33 1F 88 9F 18 99 15 90 01 13 43 04 00

0xD0: 00 00 00 00 00 00 00 00 00 00 00 0B 3B 0B 0A D2

0xE0: D1 9B 72 93 29 37 96 92 AB B9 48 96 00 00 00 00

0xF0: 00 00 00 00 00 A2 FB 14 AC 19 3D 11 54 00 C5 20

X2 Serial EEPROM Contents:

Non-Volatile Register (NVR) Fields

X2 Version :0x1E = MSA Version 0x1E

NVR Size in bytes :0x100

Number of bytes used :0x100

Basic Field Address :0xB

Customer Field Address :0x77

Vendor Field Address :0xA7

Extended Vendor Field Address :0x100

Reserved :0x0

Transceiver type :0x2 =X2

Optical connector type :0x1 =SC

Bit encoding :0x1 =NRZ

Normal BitRate in multiple of 1M b/s :0x2848

Protocol Type :0x1 =10GbE

Standards Compliance Codes :

10GbE Code Byte 0 :0x2 =10GBASE-LR

10GbE Code Byte 1 :0x0

SONET/SDH Code Byte 0 :0x0

SONET/SDH Code Byte 1 :0x0

SONET/SDH Code Byte 2 :0x0

SONET/SDH Code Byte 3 :0x0

10GFC Code Byte 0 :0x0

10GFC Code Byte 1 :0x0

10GFC Code Byte 2 :0x0

10GFC Code Byte 3 :0x0

Transmission range in 10m :0x3E8

Fibre Type :

Fibre Type Byte 0 :0x20 =SM, Generic

Fibre Type Byte 1 :0x0 =Unspecified

Centre Optical Wavelength in 0.01nm steps - Channel 0 :0x1 0xFF 0xB8

Centre Optical Wavelength in 0.01nm steps - Channel 1 :0x0 0x0 0x0

Centre Optical Wavelength in 0.01nm steps - Channel 2 :0x0 0x0 0x0

Centre Optical Wavelength in 0.01nm steps - Channel 3 :0x0 0x0 0x0

Package Identifier OUI :0xC09BE0

Transceiver Vendor OUI :0x3E800

Transceiver vendor name :CISCO-EXCELIGHT

Part number provided by transceiver vendor :SDX4102LR-C1-M

Revision level of part number provided by vendor :A

Vendor serial number :ECL1350026V

Vendor manufacturing date code :200912145A

Current Reference :

5V Stressed Environment Reference 100 percent=1A :0x0

3.3V Stressed Environment Reference 100 percent=2A :0x1

APS Stressed Environment Reference 100 percent=2A :0x2

Normal APS Voltage :0x4

Digital Optical Monitoring Capability Byte :0xC7

Digital Optical Monitoring :Implemented

Low Power Start-up(LPS) Mode Capability Bit :0x1

Reserved :0x0

Basic Field Checksum :0x18

Customer Writable Area :

0x00: 58 32 2D 31 30 47 42 2D 4C 52 20 20 20 20 20 20

0x10: 20 56 30 34 20 45 43 4C 31 33 35 30 30 32 36 56

0x20: 31 30 2D 32 30 33 36 2D 30 34 20 20 41 30 20 20

Vendor Specific :

0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x10: 00 00 00 00 00 00 00 00 00 84 01 8A 33 1F 88 9F

0x20: 18 99 15 90 01 13 43 04 00 00 00 00 00 00 00 00

0x30: 00 00 00 00 0B 3B 0B 0A D2 D1 9B 72 93 29 37 96

0x40: 92 AB B9 48 96 00 00 00 00 00 00 00 00 00 A2 FB

0x50: 14 AC 19 3D 11 54 00 C5 20

Device : X2 PMA PMD

0000: 2040 : 0010 0000 0100 0000

0001: 0006 : 0000 0000 0000 0110

0002: 0003 : 0000 0000 0000 0011

0003: E800 : 1110 1000 0000 0000

0004: 0001 : 0000 0000 0000 0001

0005: 001E : 0000 0000 0001 1110

0006: 0000 : 0000 0000 0000 0000

0007: 0006 : 0000 0000 0000 0110

0008: B145 : 1011 0001 0100 0101

0009: 0000 : 0000 0000 0000 0000

Device : X2 PCS

0000: 2040 : 0010 0000 0100 0000

0001: 0006 : 0000 0000 0000 0110

0002: 0003 : 0000 0000 0000 0011

0003: E800 : 1110 1000 0000 0000

0004: 0001 : 0000 0000 0000 0001

0005: 001E : 0000 0000 0001 1110

0006: 0000 : 0000 0000 0000 0000

0007: 0000 : 0000 0000 0000 0000

0008: 8005 : 1000 0000 0000 0101

0009: 0000 : 0000 0000 0000 0000

Device : X2 WIS

0000: 2040 : 0010 0000 0100 0000

0001: 0002 : 0000 0000 0000 0010

0002: 0003 : 0000 0000 0000 0011

0003: E800 : 1110 1000 0000 0000

0004: 0001 : 0000 0000 0000 0001

0005: 001E : 0000 0000 0001 1110

0006: 0000 : 0000 0000 0000 0000

0007: 0000 : 0000 0000 0000 0000

0008: 8003 : 1000 0000 0000 0011

0009: 0000 : 0000 0000 0000 0000

Device : X2 XGXS

0000: 2040 : 0010 0000 0100 0000

0001: 0006 : 0000 0000 0000 0110

0002: 0003 : 0000 0000 0000 0011

0003: E800 : 1110 1000 0000 0000

0004: 0001 : 0000 0000 0000 0001

0005: 001E : 0000 0000 0001 1110

0006: 0000 : 0000 0000 0000 0000

0007: 0000 : 0000 0000 0000 0000

0008: 8000 : 1000 0000 0000 0000

0009: 0000 : 0000 0000 0000 0000

Device : 10G Phy

0000: 2040 : 0010 0000 0100 0000

0001: 0080 : 0000 0000 1000 0000

0002: 0000 : 0000 0000 0000 0000

0003: 0000 : 0000 0000 0000 0000

0004: 0001 : 0000 0000 0000 0001

0005: 0020 : 0000 0000 0010 0000

0006: 0000 : 0000 0000 0000 0000

0007: 0000 : 0000 0000 0000 0000

0008: 8400 : 1000 0100 0000 0000

0009: 0000 : 0000 0000 0000 0000

C0-FF-C3, 3F-0F, 01-02

CV1-166-01-3750E#

4900 output:

"sh int te1/2 transceiver detail" is not a valid command for this device. Is there some other command I can use?

SWES-HUB-01-4900M#sh idprom int te1/2

X2 Serial EEPROM Contents:

Non-Volatile Register (NVR) Fields

X2 Version :0x1E = MSA Version 0x1E

NVR Size in bytes :0x100

Number of bytes used :0x100

Basic Field Address :0xB

Customer Field Address :0x77

Vendor Field Address :0xA7

Extended Vendor Field Address :0x100

Reserved :0x0

Transceiver type :0x2 =X2

Optical connector type :0x1 =SC

Bit encoding :0x1 =NRZ

Normal BitRate in multiple of 1M b/s :0x2848

Protocol Type :0x1 =10GbE

Standards Compliance Codes :

10GbE Code Byte 0 :0x2 =10GBASE-LR

10GbE Code Byte 1 :0x0

SONET/SDH Code Byte 0 :0x0

SONET/SDH Code Byte 1 :0x0

SONET/SDH Code Byte 2 :0x0

SONET/SDH Code Byte 3 :0x0

10GFC Code Byte 0 :0x0

10GFC Code Byte 1 :0x0

10GFC Code Byte 2 :0x0

10GFC Code Byte 3 :0x0

Transmission range in 10m :0x3E8

Fibre Type :

Fibre Type Byte 0 :0x20 =SM, Generic

Fibre Type Byte 1 :0x0 =Unspecified

Centre Optical Wavelength in 0.01nm steps - Channel 0 :0x1 0xFF 0xB8

Centre Optical Wavelength in 0.01nm steps - Channel 1 :0x0 0x0 0x0

Centre Optical Wavelength in 0.01nm steps - Channel 2 :0x0 0x0 0x0

Centre Optical Wavelength in 0.01nm steps - Channel 3 :0x0 0x0 0x0

Package Identifier OUI :0xC09BE0

Transceiver Vendor OUI :0x3E800

Transceiver vendor name :CISCO-EXCELIGHT

Part number provided by transceiver vendor :SDX4102LR-C1-M

Revision level of part number provided by vendor :A

Vendor serial number :ECL1350026D

Vendor manufacturing date code :200912145A

Current Reference :

5V Stressed Environment Reference 100 percent=1A :0x0

3.3V Stressed Environment Reference 100 percent=2A :0x1

APS Stressed Environment Reference 100 percent=2A :0x2

Normal APS Voltage :0x4

Digital Optical Monitoring Capability Byte :0xC7

Digital Optical Monitoring :Implemented

Low Power Start-up(LPS) Mode Capability Bit :0x1

Reserved :0x0

Basic Field Checksum :0x6

Customer Writable Area :

0x00: 58 32 2D 31 30 47 42 2D 4C 52 20 20 20 20 20 20

0x10: 20 56 30 34 20 45 43 4C 31 33 35 30 30 32 36 44

0x20: 31 30 2D 32 30 33 36 2D 30 34 20 20 41 30 20 20

Vendor Specific :

0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x10: 00 00 00 00 00 00 00 00 00 84 01 8A 33 1F 88 9F

0x20: 18 99 15 90 01 13 43 04 1F 00 00 00 00 00 00 00

0x30: 00 00 00 00 0B 6D 7E 22 F3 D4 CC 63 9D 00 03 89

0x40: 9A A2 0D 94 2C 00 00 00 00 00 00 00 00 00 A2 65

0x50: EB B7 16 7B 11 40 00 D8 20

SWES-HUB-01-4900M#

InayathUlla Sharieff · ‎04-08-2013

Hi Span,

Probably you are hitting the following bug:

CSCts58073 Bug Details

Next Action: Kindly upgrade the 4900 device to code 12.2(55) SE5 which seems to have this error resolved.

HTH

Regards

Inayath

*Plz rate the usefull posts.

Bilal Nawaz · ‎04-08-2013

Hello, as mentioned this is a IOS bug
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCts58073
Upgrade the 3750e to
15.0(2)SE or
12.2(55)SE5
where apparently gets fixed.

So I assume your 10Gb interface is up?

however, it doesn't explain why several ports are going in to an error disabled state. Are these trunked ports or access ports? And where do they connect to? (This behaviour doesn't seem to be a symptom of the bug)

Hope this helps.

Sent from Cisco Technical Support iPhone App

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

spaansj05 · ‎04-09-2013

They are all access ports in our campus dormatories.

At first I thought maybe the students were plugging the same malfunctioning device or bad cable into the ports but I have verified that they have tried several different devices and cables. We have never had an issue with these data drops and switchports before this.

Bilal Nawaz · ‎04-09-2013

Okay there may be a few reasons why..

Could you show us the configuration on these access ports please?

You should be able to see the cause if you were to console on to the switch; or even ssh/telnet to it with 'term mon', and you plug in to an access port, you should see it come up with a reason if it goes to err-disable?

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml

Causes of Errdisable

This feature was first implemented to handle special collision situations in which the switch detected excessive or late collisions on a port. Excessive collisions occur when a frame is dropped because the switch encounters 16 collisions in a row. Late collisions occur after every device on the wire should have recognized that the wire was in use. Possible causes of these types of errors include:

A cable that is out of specification (either too long, the wrong type, or defective)
A bad network interface card (NIC) card (with physical problems or driver problems)
A port duplex misconfigurationA port duplex misconfiguration is a common cause of the errors because of failures to negotiate the speed and duplex properly between two directly connected devices (for example, a NIC that connects to a switch). Only half-duplex connections should ever have collisions in a LAN. Because of the carrier sense multiple access (CSMA) nature of Ethernet, collisions are normal for half duplex, as long as the collisions do not exceed a small percentage of traffic.

There are various reasons for the interface to go into errdisable. The reason can be:

Duplex mismatch
Port channel misconfiguration
BPDU guard violation
UniDirectional Link Detection (UDLD) condition
Late-collision detection
Link-flap detection
Security violation
Port Aggregation Protocol (PAgP) flap
Layer 2 Tunneling Protocol (L2TP) guard
DHCP snooping rate-limit
Incorrect GBIC / Small Form-Factor Pluggable (SFP) module or cable
Address Resolution Protocol (ARP) inspection
Inline power

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.