07-15-2013 01:47 AM - edited 03-07-2019 02:24 PM
Hi ,
I have cisco WS-C2960S switches at the access layer spread around my network , I noticed lately that many ports get disconnected at random
this is the log :
024610: Jul 14 09:41:33: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024611: Jul 14 09:41:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
024612: Jul 14 09:41:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
024613: Jul 14 09:41:55: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down
024614: Jul 14 09:41:58: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024615: Jul 14 09:41:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
024616: Jul 14 09:42:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
024617: Jul 14 09:42:18: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down
024618: Jul 14 09:42:21: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024619: Jul 14 09:42:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
024620: Jul 14 09:45:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
024621: Jul 14 09:45:32: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down
024622: Jul 14 09:45:40: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024623: Jul 14 09:45:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
024624: Jul 14 09:45:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
024625: Jul 14 09:45:59: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down
024626: Jul 14 09:46:02: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024627: Jul 14 09:46:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
024853: Jul 15 09:54:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
024854: Jul 15 09:54:55: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down
024855: Jul 15 09:54:57: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up
024856: Jul 15 09:54:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
I checked configuration and infratsructure , No problem there , I cant say exactly when it started but no major changes were made in the past few months.
we also have a NAC appliance from "Forescout" which shuts down ports for security violations but the log doesnt state Admin Down , its as if
the port gets disconnected , its not Flapping otherwise it would trigger an err-disabled state , connected to the ports are computers , IP-Phones
and various equipment
I am currently running
12.2(55)SE3
and will upgrade one Stack just to rull it out , anybody have any Idea what kind of tests I can do to fugure out where this is coming from?
thanks
07-15-2013 09:32 AM
Probably a layer 1 issue.
Try changing the cables. Testing/changing the patch cables and patch panel run through the wall. Or change the end device. See if the problem stays to the same port of if the problem travles with the end device.
07-15-2013 10:32 AM
Its not a layer 1 issue. Its happening all across the network randomly, i have tried changing the cables and infrastructure and it didnt help. Also tried hard coding speed and duplex.
I Know that judging by the logs it could only be a layer one issue but I am afraid its not that simple. My only explanation is that the NAC security appliance is causing the problem even though its not putting the port in a shutdown state. Has anyone seen this kind of thing happen?
Sent from Cisco Technical Support Android App
07-15-2013 11:03 AM
What port states are your ports getting put into?
Also is this only happening on Access ports that are stipulated by your NAC?
07-15-2013 04:52 PM
Its not a layer 1 issue. Its happening all across the network randomly,
Ok, do this:
1. Command: test cable tdr interface Gi2/0/39;
2. Wait for about 5 to 7 seconds;
3. Command: sh cable tdr interface Gi2/0/39; and
4. Post the output to #3.
07-15-2013 02:10 PM
Normally when a computer is in some kind of violation the NAC tells the switch to shut down the port. This may not be the case because the ports are not admin down, they are simply disconnected, and flap a few times every minute like the log shows. I took one switch and cut off all NAC access to it but still didnt help. The NAC appliance can still learn about the computer from arp table of aggregation switch and investigate the computer itself through nmap and other tools. But I want to know if that sort of activity can cause the computer to disconnect its network connection regardless to the state of the port on the switch.
Sent from Cisco Technical Support Android App
07-16-2013 09:44 PM
Have you checked the pcs on the port for power saving features? Pcs will "sleep" and wake frequently in power saving mode and I've seen this type of frequent flapping on associated ports.
Sent from Cisco Technical Support iPad App
07-22-2013 06:37 AM
hi all thanks for all the replyes
here is output of the test cable command :
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi2/0/39 1000M Pair A 39 +/- 0 meters Pair A Normal
Pair B 37 +/- 0 meters Pair B Normal
Pair C 38 +/- 0 meters Pair C Normal
Pair D 38 +/- 0 meters Pair D Normal
I also mapped all computers connected to one switch , Hardcoded their speed&Duplex and changed the settings of the
NIC so that the computer cant disable the NIC . I will give it a few days and see how it goes.
today I am about to upgrade a different switch just to rull out a bug.
will let you know , by the way I heard there is a way to troubleshoot such a problem with a sniffer , anybody
know what I should be looking for after I sniffed the traffic ?
thanks
07-22-2013 03:50 PM
TDR shows clear.
Can you post the output to the following commands:
1. sh interface G2/0/39; and
2. sh controller ether G2/0/39
07-22-2013 09:31 AM
Hello Fisher,
Not sure if those notifications are from the console or telnet interface. If it is from telnet you may want to monitor the console interface to see a more descriptive output. Also check out this link: http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml
I was having a similar issue with a prot turning off ar random intervals. I then watched first hand it error out on the console messages.
Regards,
Eddie
08-01-2013 02:12 PM
07-22-2013 04:38 PM
hello,
im curious and have to ask are you running rip versiion 2, if so it sounds like you could hve a routing loop maybe. not 100 percent sure but it is a suggestion.
08-05-2013 01:27 PM
Thanks everybody for all the good stuff, you opened my mind for things i didnt know. After some troubleshooting we found no problems network related. We have concluded that there is no issue here, the so called problem has to be caused by the power management of the computers.
Thanks everybody for your time.
Sent from Cisco Technical Support Android App
03-25-2019 01:21 PM
Could you please explain the problem. I have the same
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide