Here in my office, we are all scratching our heads on this one.
Switch A Switch B
So we have two switches, one is CORE Switch A, the other is a Distro Switch B, a 3750-G switch. A and B are connected via a 2 link Etherchannel via Port Channel 1 (on both switches). Each switch's ports are configured the same:
no ip address
channel-group 1 mode active
no spanning-tree portfast
Fiber is used between the devices.
UDLD aggressive was on globally on the switch
The port channels are configured as:
ip address 10.10.10.208 255.255.255.254 10.10.10.209 255.255.255.254 (everthing else same on B)
no ip proxy-arp
no ip redirects
So we had one of the links, G1/0/1, drop. Doing a sho int status was showing G1/0/1 "not connected" on both switches. Our focus began to form on issues on Switch B, the distro switch. There is a lot of stuff I can't disclose, but the CORE switch seems to be out of the loop on this. Anyway...
So testing of the fiber/port started with a loopback being placed in the port G1/0/1. This caused the port to err-disable. The port light went amber.
We reset the port; port still went amber.
We removed the loopback and reset the switch; port stayed amber
We shut the port totally; port would show not connected, but the amber light remained on.
We tried clearing port-security, no change. Reset UDLD, no change. Removed UDLD aggressive, no change. Note that STP was not putting these in blocking state.
We then tried putting the same configs in different ports. The same issues would come up...err-disable, we'd shut/no shut, but the amber port light would come on and remain. This happened on all three switches in the stack. We are stumped.
What is causing these port lights to come on and remain amber?
Only a hard reboot of the switch cleared the amber lights. But once again, the problem returned.
This caused the port to err-disable. The port light went amber.
WHY did the port go into err-disable? Post the output to the command "sh interface status err".
If you remove the etherchannel configuration, does all the port go up/up?
1. Since the switch was hard-rebooted, I dont have any results for the sh interface status err. The ports are currently not-connected.
2. When we removed the etherchannel config, the port remained down and the amber light remained on. I took one of the ports, shut it, took the entire config out of it. Then, I entered it as switchport mode trunk, switchport encapulation dot1q.
We then tried a loopback on it; as soon as I did "no shut", it err-disabled.
The errors were (cant copy/post the header for reasons)
udld...udld_port_disable: UDLD disabled port g1/0/1, transmit/receive loop detected
SP4...UDLD error detected in G1/0/1, putting in err-disable mode
This occurred both before UDLD aggressive was off, and after. It also occurred when nothing plugged into the port and we turned the port back on. The thing is, it did this on three separate switches with several different ports in each switch.
transmit/receive loop detected
I believe this means that the port is receiving the same "Hello" packet being sent out.
Let's make things simple.
1. Disable UDLD on those ports FIRST.
2. Configure etherchannel;
3. Is the etherchannel up?
4. Are all ports in the etherchannel active?
5. Enable UDLD
6. Check #3 & #4.
Unfortunatly, I can't copy/paste as this is on another network.
And, the wire guys went out and did some stuff...so I can't confirm that original ports are reconnected. I'll come in tonight and see where things stand. THanks all, we'll see how it goes.
Follow-up to this issue:
We found out a couple things during troubleshooting. If you put a loopback on the port, the port will go amber. It will remain amber until the switch is restarted. However, putting in a "good" link will clear the port of the amber status.
In our scenario, the wire guys were using the "bad" link and testing it on several different ports. This caused all the ports to go amber. When we restarted the switch, it cleared it up EXCEPT where we had the bad link. This was some good info to learn.
Bottom line, we had a bad fiber pair. This could have been avoided if the wire guys shot light down the fiber first, but they are in a different group and as such, I have no control over them
They replaced the fiber, and both links on the port channel came up and all is well.