Cisco 3750-G Amber port lights remain lit, nothing in port

[NormMuelleman]

Here in my office, we are all scratching our heads on this one.

Switch A                                             Switch B

Po1                                                       Po1

G1/0/1                                                  G1/0/1

G2/0/1                                                  G2/0/1

So we have two switches, one is CORE Switch A, the other is a Distro Switch B, a 3750-G switch. A and B are connected via a 2 link Etherchannel via Port Channel 1 (on both switches). Each switch's ports are configured the same:

no switchport

no ip address

channel-protocol lacp

channel-group 1 mode active

no spanning-tree portfast

Fiber is used between the devices.

UDLD aggressive was on globally on the switch

The port channels are configured as:

no switchport

ip address 10.10.10.208 255.255.255.254     10.10.10.209 255.255.255.254 (everthing else same on B)

no ip proxy-arp

no ip redirects

So we had one of the links, G1/0/1, drop. Doing a sho int status was showing G1/0/1 "not connected" on both switches. Our focus began to form on issues on Switch B, the distro switch. There is a lot of stuff I can't disclose, but the CORE switch seems to be out of the loop on this. Anyway...

So testing of the fiber/port started with a loopback being placed in the port G1/0/1. This caused the port to err-disable. The port light went amber.

We reset the port; port still went amber.

We removed the loopback and reset the switch; port stayed amber

We shut the port totally; port would show not connected, but the amber light remained on.

We tried clearing port-security, no change. Reset UDLD, no change. Removed UDLD aggressive, no change. Note that STP was not putting these in blocking state.

We then tried putting the same configs in different ports. The same issues would come up...err-disable, we'd shut/no shut, but the amber port light would come on and remain. This happened on all three switches in the stack. We are stumped.

What is causing these port lights to come on and remain amber?

Only a hard reboot of the switch cleared the amber lights. But once again, the problem returned.

[Leo Laohoo]

This caused the port to err-disable. The port light went amber.

WHY did the port go into err-disable?  Post the output to the command "sh interface status err".

If you remove the etherchannel configuration, does all the port go up/up?

[NormMuelleman]

1. Since the switch was hard-rebooted, I dont have any results for the sh interface status err. The ports are currently not-connected.

2. When we removed the etherchannel config, the port remained down and the amber light remained on. I took one of the ports, shut it, took the entire config out of it. Then, I entered it as switchport mode trunk, switchport encapulation dot1q.

We then tried a loopback on it; as soon as I did "no shut", it err-disabled.

The errors were (cant copy/post the header for reasons)

udld...udld_port_disable: UDLD disabled port g1/0/1, transmit/receive loop detected

SP4...UDLD error detected in G1/0/1, putting in err-disable mode

This occurred both before UDLD aggressive was off, and after. It also occurred when nothing plugged into the port and we turned the port back on. The thing is, it did this on three separate switches with several different ports in each switch.

[Leo Laohoo]

transmit/receive loop detected

I believe this means that the port is receiving the same "Hello" packet being sent out.

Let's make things simple.

1. Disable UDLD on those ports FIRST. 

2. Configure etherchannel;

3.  Is the etherchannel up?

4.  Are all ports in the etherchannel active?

5.  Enable UDLD

6.  Check #3 & #4.

Try that.

[Leo Laohoo]

You rebooted the switch recently?  Perfect.  Also post the output to the command "sh post".

Can you also post the output to the command "sh interface "?

[NormMuelleman]

Unfortunatly, I can't copy/paste as this is on another network.

And, the wire guys went out and did some stuff...so I can't confirm that original ports are reconnected. I'll come in tonight and see where things stand. THanks all, we'll see how it goes.

[NormMuelleman]

Follow-up to this issue:

We found out a couple things during troubleshooting. If you put a loopback on the port, the port will go amber. It will remain amber until the switch is restarted. However, putting in a "good" link will clear the port of the amber status.

In our scenario, the wire guys were using the "bad" link and testing it on several different ports. This caused all the ports to go amber. When we restarted the switch, it cleared it up EXCEPT where we had the bad link. This was some good info to learn.

Bottom line, we had a bad fiber pair. This could have been avoided if the wire guys shot light down the fiber first, but they are in a different group and as such, I have no control over them

They replaced the fiber, and both links on the port channel came up and all is well.