3750 switch interface crc/error problem

andrewswanson · ‎08-13-2012

Hello

I'm having a problem with a WS-C3750G-12S-E. I'm seeing CRCs/input errors only on specific interfaces on the 3750. Initially i thought this was a fibre issue so i cleaned/replaced the fibre patch leads as well as the SFPs at both ends. the CRC's/errors continued until i removed the SFP from the switch and placed it into another port on the same switch.

if i use interfaces gi1/0/1, 2, 5, 6, 9 or 10 i get errors/CRCs - if i move the SFP (fibre or copper) to interfaces gi1/0/3, 4, 7, 8, 11, or 12 the errors/CRCs stop. The output of "sh platform pm if-numbers" below shows that the switch has 3 asics (with 4 ports) - the interfaces i'm having a problem with are spread across all 3 asics but only on the last 2 ports (2 and 3).

interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb

----------------------------------------------------------------------

Gi1/0/1 1 1 1 2/2 1 1 1 local Yes Yes

Gi1/0/2 2 2 2 2/3 1 2 2 local Yes Yes

Gi1/0/3 3 3 3 2/0 1 3 3 local Yes Yes

Gi1/0/4 4 4 4 2/1 1 4 4 local Yes Yes

Gi1/0/5 5 5 5 0/2 1 5 5 local Yes Yes

Gi1/0/6 6 6 6 0/3 1 6 6 local Yes Yes

Gi1/0/7 7 7 7 0/0 1 7 7 local Yes Yes

Gi1/0/8 8 8 8 0/1 1 8 8 local Yes Yes

Gi1/0/9 9 9 9 1/2 1 9 9 local Yes Yes

Gi1/0/10 10 10 10 1/3 1 10 10 local Yes Yes

Gi1/0/11 11 11 11 1/0 1 11 11 local Yes Yes

Gi1/0/12 12 12 12 1/1 1 12 12 local Yes Yes

there is very little traffic through the switch and cpu is ok. Could this be a hardware fault? switch is currently running c3750-ipservicesk9-mz.122-55.SE6.bin ( upgraded from 52). output of the command sh platform port-asic stats drop for the affected interfaces show zero drops.

thanks in advance

andy

Alessio Andreoli · ‎08-13-2012

Try this :

http://packetlife.net/blog/2008/mar/28/investigating-ciscos-built-in-tdr/

and anyway i would check how duplex and speed are configured. they should be hardcoded and with no negotiation.

Alessio

andrewswanson · ‎08-13-2012

thanks for the resonse Alessio - unfortunately tdr isn't support on sfp ports ( i get % TDR test is not supported on interface ) when i try. I've already double checked speed/duplex settings and all are ok. as i say, the problem is resolved when i move the sfp onto gi1/0/3,4,7,8,11,12 but i don't have enough of these 'working' ports for all my links.

thanks

andy

Leo Laohoo · ‎08-13-2012

Hi Andrew,

Can you please post the following output:

1. sh version;

2. sh controller e Gi ; and

3. sh interface Gi

Leo Laohoo · ‎08-13-2012

Hi Alessio,

Cisco switches will support copper TDR but OTDR is not yet supported.

andrewswanson · ‎08-14-2012

hi leolaohoo - requested output below (for gi1/0/1 - one of the affected ports).

thanks

andy

sh ver
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(55)SE6,
RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Mon 23-Jul-12 13:22 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE
(fc1)

Mer3750SOC-C31-1 uptime is 1 day, 3 hours, 9 minutes
System returned to ROM by power-on
System restarted at 06:01:27 BST Mon Aug 13 2012
System image file is "flash:c3750-ipservicesk9-mz.122-55.SE6.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3750G-12S (PowerPC405) processor (revision H0) with 131072K bytes of m
emory.
Processor board ID CAT0828R0R2
Last reset from power-on
7 Virtual Ethernet interfaces
12 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:11:93:25:3C:00
Motherboard assembly number     : 73-8307-08
Power supply part number        : 341-0048-01
Motherboard serial number       : CAT08270HUH
Power supply serial number      : DTH082011GM
Model revision number           : H0
Motherboard revision number     : A0
Model number                    : WS-C3750G-12S-E
System serial number            : CAT0828R0R2
Top Assembly Part Number        : 800-23419-01
Top Assembly Revision Number    : K0
Version ID                      : N/A
Hardware Board Revision Number : 0x07

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 12    WS-C3750G-12S      12.2(55)SE6           C3750-IPSERVICESK9-M

Configuration register is 0xF

sh controllers ethernet-controller gi1/0/1

     Transmit GigabitEthernet1/0/1            Receive
    259938503 Bytes                       1207273348 Bytes
       736997 Unicast frames                 1034568 Unicast frames
       320591 Multicast frames                214721 Multicast frames
       198162 Broadcast frames                 20997 Broadcast frames
            0 Too old frames              1184134313 Unicast bytes
            0 Deferred frames               16087093 Multicast bytes
            0 MTU exceeded frames            1724812 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames               52012 Minimum size frames
            0 8 collision frames              371239 65 to 127 byte frames
            0 9 collision frames               43419 128 to 255 byte frames
            0 10 collision frames              34665 256 to 511 byte frames
            0 11 collision frames              14625 512 to 1023 byte frames
            0 12 collision frames              32619 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions               730 Symbol error frames
            0 Late collisions                    119 Invalid frames, too large
            0 VLAN discard frames             722288 Valid frames, too large
            0 Excess defer frames                 31 Invalid frames, too small
         4033 64 byte frames                       0 Valid frames, too small
       943917 127 byte frames
       129003 255 byte frames                      0 Too old frames
        56374 511 byte frames                      1 Valid oversize frames
        40047 1023 byte frames                     0 System FCS error frames
        79734 1518 byte frames                     0 RxPortFifoFull drop frame
         2642 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

sh interfaces gi1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0011.9325.3c01 (bia 0011.9325.3c01)
Description:
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4000 bits/sec, 2 packets/sec
5 minute output rate 7000 bits/sec, 6 packets/sec
     1270755 packets input, 1207360054 bytes, 0 no buffer
     Received 235952 broadcasts (214916 multicasts)
     0 runts, 1 giants, 0 throttles
     730 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 214916 multicast, 0 pause input
     0 input packets with dribble condition detected
     1256434 packets output, 260085515 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎08-15-2012

Thanks for the output. Really helps.

Your line errors are "Symbol errors" being received by the interface.

May I ask what is connected to this?

if i use interfaces gi1/0/1, 2, 5, 6, 9 or 10 i get errors/CRCs - if i move the SFP (fibre or copper) to interfaces gi1/0/3, 4, 7, 8, 11, or 12 the errors/CRCs stop.

This description is very "disturbing". I believe the 3750 has a single 3 ASICs. And if you say that moving the SFP to 3 & 4, ports 7 & 8 and ports 11 & 12 and the line errors stops, I am inclined to believe that the ASIC is failing.