access-ports randomly Flapping

[nir.fisher]

Hi ,

I have cisco WS-C2960S switches at the access layer spread around my network , I noticed lately that many ports get disconnected at random

this is the log :

024610: Jul 14 09:41:33: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024611: Jul 14 09:41:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

024612: Jul 14 09:41:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

024613: Jul 14 09:41:55: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down

024614: Jul 14 09:41:58: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024615: Jul 14 09:41:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

024616: Jul 14 09:42:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

024617: Jul 14 09:42:18: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down

024618: Jul 14 09:42:21: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024619: Jul 14 09:42:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

024620: Jul 14 09:45:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

024621: Jul 14 09:45:32: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down

024622: Jul 14 09:45:40: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024623: Jul 14 09:45:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

024624: Jul 14 09:45:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

024625: Jul 14 09:45:59: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down

024626: Jul 14 09:46:02: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024627: Jul 14 09:46:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

024853: Jul 15 09:54:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

024854: Jul 15 09:54:55: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to down

024855: Jul 15 09:54:57: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/39, changed state to up

024856: Jul 15 09:54:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up

I checked configuration and infratsructure , No problem there , I cant say exactly when it started but no major changes were made in the past few months.

we also have a NAC appliance from "Forescout" which shuts down ports for security violations but the log doesnt state Admin Down , its as if

the port gets disconnected , its not Flapping otherwise it would trigger an err-disabled state , connected to the ports are computers , IP-Phones

and various equipment

I am currently running

12.2(55)SE3

and will upgrade one Stack just to rull it out , anybody have any Idea what kind of tests I can do to fugure out where this is coming from?

thanks

[mikegrous]

Probably a layer 1 issue.

Try changing the cables. Testing/changing the patch cables and patch panel run through the wall. Or change the end device. See if the problem stays to the same port of if the problem travles with the end device.

[nir.fisher]

Its not a layer 1 issue. Its happening all across the network randomly, i have tried changing the cables and infrastructure and it didnt help. Also tried hard coding speed and duplex.
I Know that judging by the logs it could only be a layer one issue but I am afraid its not that simple. My only explanation is that the NAC security appliance is causing the problem even though its not putting the port in a shutdown state. Has anyone seen this kind of thing happen?


Sent from Cisco Technical Support Android App

[Hameed Shah]

What port states are your ports getting put into?   

Also is this only happening on Access ports that are stipulated by your NAC?

[Leo Laohoo]

Its not a layer 1 issue. Its happening all across the network randomly,

Ok, do this:

1.  Command:  test cable tdr interface Gi2/0/39;

2.  Wait for about 5 to 7 seconds;

3.  Command:  sh cable tdr interface Gi2/0/39; and

4.  Post the output to #3.

[nir.fisher]

Normally when a computer is in some kind of violation the NAC tells the switch to shut down the port. This may not be the case because the ports are not admin down, they are simply disconnected, and flap a few times every minute like the log shows. I took one switch and cut off all NAC access to it but still didnt help. The NAC appliance can still learn about the computer from arp table of aggregation switch and investigate the computer itself through nmap and other tools. But I want to know if that sort of activity can cause the computer to disconnect its network connection regardless to the state of the port on the switch.


Sent from Cisco Technical Support Android App

[Jeff Van Houten]

Have you checked the pcs on the port for power saving features? Pcs will "sleep" and wake frequently in power saving mode and I've seen this type of frequent flapping on associated ports.

Sent from Cisco Technical Support iPad App

[nir.fisher]

hi all thanks for all the replyes

here is output of the test cable command :

Interface Speed Local pair Pair length        Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Gi2/0/39  1000M Pair A     39   +/- 0  meters Pair A      Normal             

                         Pair B     37   +/- 0  meters Pair B      Normal             

                         Pair C     38   +/- 0  meters Pair C      Normal             

                         Pair D     38   +/- 0  meters Pair D      Normal    

I also mapped all computers connected to one switch , Hardcoded their speed&Duplex and changed the settings of the

NIC so that the computer cant disable the NIC . I will give it a few days and see how it goes.    

today I am about to upgrade a different switch just to rull out a bug.

will let you know , by the way I heard there is a way to troubleshoot such a problem with a sniffer , anybody

know what I should be looking for after I sniffed the traffic ?

thanks

[Leo Laohoo]

TDR shows clear.

Can you post the output to the following commands:

1.  sh interface G2/0/39; and

2.  sh controller ether G2/0/39

[eddiebrown1986]

Hello Fisher,

Not sure if those notifications are from the console or telnet interface. If it is from telnet you may want to monitor the console interface to see a more descriptive output. Also check out this link: http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml

I was having a similar issue with a prot turning off ar random intervals. I then watched first hand it error out on the console messages.

Regards,

Eddie

[Hameed Shah]

chris marcocci, From my understanding He is talking about access layer switches "around his environment", from a topological standpoint it would not be feasible to run routing protocols from his access layer switches......Especially since they are all Line up/down....It seems to be some type of L2 mechanism at play here

[chr6is000]

hello,

im curious and have to ask are you running rip versiion 2, if so it sounds like you could hve a routing loop maybe. not 100 percent sure but it is a suggestion.

[nir.fisher]

Thanks everybody for all the good stuff, you opened my mind for things i didnt know. After some troubleshooting we found no problems network related. We have concluded that there is no issue here, the so called problem has to be caused by the power management of the computers.
Thanks everybody for your time.


Sent from Cisco Technical Support Android App

[Added Solucoes]

Could you please explain the problem. I have the same