Help Troubleshooting 3750G-12S stack Interface connections

Andrew Bailey · ‎04-06-2011

Here is my problem.... Thanks in advance for any help!

I have 2 Cisco WS-C3750G-12S fiber switches in a stack, the first 11 interfaces in each switch are configured the same, port 12 in each switch is in a port channel. Connected to the switch are bunch of desktops running Windows and Linux with fiber NICs.

Here is an example from an interface on switch 2 of the config on the first 11 ports of each switch:

interface GigabitEthernet2/0/1

description Port_Description

switchport access vlan 2

no cdp enable

spanning-tree portfast

On switch 1, (master switch), everything is working fine, all the interfaces are up.

On switch 2, interfaces 3-4, 7-8, 11-12, work fine, no problems. However Interfaces 1-2, 5-6, 9-10 do not come up.

I tested the dead interfaces (Interface GigabitEthernet2/0/1 for example) with multiple known good SFPs from one of the working ports as well as swapping the network connection from a working desktop (Interface GigabitEthernet2/0/3) and I get no link. Status of the switch interface GigabitEthernet2/0/1 is down, line protocol is down (notconnect). It remains the same after a shutdown and no shutdown applied.

Next, I put a 1GB copper SFP in Interface GigabitEthernet2/0/1 and connected a laptop with 1000Mbps NIC. The interface comes right up.

I performed the same tests on the other dead interfaces with the same result.

So from my troubleshooting it appears that there is not a problem with

Cabling path to workstation
SFP
Switchport interface

Am I missing something else? I seem to have eliminated every possible cause of the interfaces not coming up... the SFP, network connection to desktop, and switch interface. Is there anything else to test? What could the problem be?

Leo Laohoo · ‎04-06-2011

Can you please post the following output to the commands:

1. sh version (I want to see how long has both switches been up and I want to see what IOS you're running)

2. sh post (If you get a FAIL then you need to RMA the switch)

Andrew Bailey · ‎04-07-2011

Ok here is the output from show post and show version while consoled into the master switch. I am not sure why show version is only showing Switch 2. We have 1 stack cable connecting switch 1 port 1 (master switch) to switch 2 port 2 (member switch).

___________________________________

sh post

Stored system POST messages:

Switch 1

---------

POST: CPU MIC register Tests : Begin

POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin

POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin

POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin

POST: PortASIC RingLoopback Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin

POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin

POST: Found Stack port 2 Down

POST: PortASIC Stack Port Loopback Tests : End, Status Passed

--More--

POST: PortASIC Port Loopback Tests : Begin

POST: PortASIC Port Loopback Tests : End, Status Passed

Switch 2

---------

POST: CPU MIC register Tests : Begin

POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin

POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin

POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin

POST: PortASIC RingLoopback Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin

POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin

--More-- POST: Found Stack port 1 Down

POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin

POST: PortASIC Port Loopback Tests : End, Status Passed

switch#sh ver

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 21-Apr-10 04:49 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc2)

switch uptime is 1 week, 1 day, 18 hours, 37 minutes

System returned to ROM by power-on

System image file is "flash:c3750-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

--More-- to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750G-12S (PowerPC405) processor (revision P0) with 131072K bytes of memory.

Processor board ID FHK1115X02G

Last reset from power-on

2 Virtual Ethernet interfaces

24 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:1B:8F:78:3E:80

Motherboard assembly number : 73-9678-07

Power supply part number : 341-0048-03

Motherboard serial number : CAT11125N8G

Power supply serial number : DTH1042D12K

Model revision number : P0

--More-- Motherboard revision number : A0

Model number : WS-C3750G-12S-S

System serial number : FHK1115X02G

Top Assembly Part Number : 800-25856-03

Top Assembly Revision Number : A0

Version ID : V05

CLEI Code Number : CNM81V0GRA

Hardware Board Revision Number : 0x06

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 12 WS-C3750G-12S 12.2(53)SE2 C3750-IPSERVICESK9-M

2 12 WS-C3750G-12S 12.2(53)SE2 C3750-IPSERVICESK9-M

Switch 02

---------

Switch Uptime : 1 week, 1 day, 0 minutes

Base ethernet MAC Address : 00:11:BB:C0:16:80

Motherboard assembly number : 73-8307-08

Power supply part number : 341-0048-01

Motherboard serial number : CAT083305PT

--More-- Power supply serial number : DTH0825176G

Model revision number : H0

Motherboard revision number : A0

Model number : WS-C3750G-12S-E

System serial number : CAT0833Z0QA

Top assembly part number : 800-23419-01

Top assembly revision number : K0

Version ID : N/A

Configuration register is 0xF

Leo Laohoo · ‎04-07-2011

--More-- POST: Found Stack port 1 Down

Thanks for the post.

That's odd. Can you post the output for "sh log" and "sh switch detail" please?

Andrew Bailey · ‎04-08-2011

There is nothing in the log regarding the problematic member switch or it's interfaces 2/0/1 - 2/0/12. The log does contain interface state changes from the master switch, I am not sure why there aren't any from the member switch.

sh switch detail

Switch/Stack Mac Address : 001b.8f78.3e80

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

*1 Master 001b.8f78.3e80 15 0 Ready

2 Member 0011.bbc0.1680 1 0 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Down 2 None

2 Down Ok None 1

glen.grant · ‎04-06-2011

You could try to reload the 2nd switch in the stack and see what happens.

Andrew Bailey · ‎04-08-2011

Yes, we plan to reload it, but we need to do it after hours and haven't had a chance yet.

Leo Laohoo · ‎04-08-2011

Yes, we plan to reload it, but we need to do it after hours and haven't had a chance yet.

Someone could "trip" over the power cable.

Andrew Bailey · ‎04-12-2011

I have some more information that makes this problem even more strange...

The member switch in the stack that is giving us problems is not the original switch in the stack. We had some issues with another identical 3750 and swapped it out. I was testing the original switch in standalone to confirm it was working. It is doing the EXACT same thing!

Interfaces 3-4, 7-8, 11-12, work fine, but Interfaces 1-2, 5-6, 9-10 do not come up. Interfaces have the same config, tested with known good SFP and link to another switch.

So I have two switches that are doing the same thing... one is running in a stack, one standalone.

We thought it could be a hardware issue with the asics, but half the ports in each asic work and half don't.

Switch#sh platform pm platform-block

interface gid gpn lpn asic hw-i flags sp dp bundle vlan mvid mac so_di i_vlan

-----------------------------------------------------------------------------

Gi1/0/1 1 1 1 2 2 U 3 2 no 0 0 0 61441 0

Gi1/0/2 2 2 2 2 3 D 0 0 no 0 0 0 61442 0

Gi1/0/3 3 3 3 2 0 D 3 2 no 0 0 0 61443 0

Gi1/0/4 4 4 4 2 1 D 0 0 no 0 0 0 61444 0

Gi1/0/5 5 5 5 0 2 D 0 0 no 0 0 0 61445 0

Gi1/0/6 6 6 6 0 3 D 0 0 no 0 0 0 61446 0

Gi1/0/7 7 7 7 0 0 D 3 2 no 0 0 0 61447 0

Gi1/0/8 8 8 8 0 1 D 0 0 no 0 0 0 61448 0

Gi1/0/9 9 9 9 1 2 D 0 0 no 0 0 0 61449 0

Gi1/0/10 10 10 10 1 3 D 0 0 no 0 0 0 61450 0

Gi1/0/11 11 11 11 1 0 D 3 2 no 0 0 0 61451 0

Gi1/0/12 12 12 12 1 1 D 0 0 no 0 0 0 61452 0

Andrew Bailey · ‎04-13-2011

Called Cisco TAC and concluded we have a hardware issue... they are shipping us replacement switches.