Port Disables Every 10 Minutes - Page 2

TrustNetworks · ‎08-16-2012

Good Afternoon

I am experiencing the same problem described in this post https://supportforums.cisco.com/message/3421454#3421454

I have seen this happen on different networks, with different equipment attached. It happens on both 2960 and 3750 switches.

Basically the connection drops, and we see in the web interface "Port is Disabled". This appears to happen every 10 minutes.

On the CLI, the status shows as connected.

Port Name Status Vlan Duplex Speed Type

Fa0/38 connected 1 a-full a-100 10/100BaseTX

I have ran cable diagnostics while the drop out is occuring.

Interface Speed Local pair Pair length Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Fa0/38 100M Pair A 28 +/- 15 meters Pair B Normal

Pair B 28 +/- 15 meters Pair A Normal

Pair C N/A Pair D Not Supported

Pair D N/A Pair C Not Supported

During the outage, I see the duplex flucate between full and half. The outage occurs for approx 90 seconds

If I fix the duplex and speed at both ends, the outage reduces to around 30 seconds.

If I apply spanning-tree portfast the outage reduces further to around 10 seconds.

Before I change any configuration on the port, the logs show the interface going down

Aug 16 13:06:51.875: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/38, changed state to down

Aug 16 13:06:52.874: %LINK-3-UPDOWN: Interface FastEthernet0/38, changed state to down

Aug 16 13:06:55.306: %LINK-3-UPDOWN: Interface FastEthernet0/38, changed state to up

Aug 16 13:06:56.313: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/38, changed state to up

Aug 16 13:07:28.089: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/38, changed state to down

Aug 16 13:07:30.094: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/38, changed state to up

However, once I apply the configuration nothing is logged. However we can still see the conneciton is disappearing for around 10 seconds.

I suspect the issue wasn't resolved for the person reporting the problem in the link above, but because the outage is minimized, and not being logged it is going unnoticed.

Does anyone know what causes this, or how I can troubleshoot further.

Regards

Paul

burleyman · ‎08-22-2012

Forgot one other question.....do some other these connect through a IP phone?

Mike

burleyman · ‎08-22-2012

Ok brain a little slow..another question...... do these things have wake on LAN turned on? if so turn it off. This has caused us similar issues with printers and servers when we mistakenly left it on.

Mike

TrustNetworks · ‎08-22-2012

Hi Mike

The errors aren't increasing, the ones shown in the stats were occurring during the duplex/speed negotiations. On this particular port, I fixed the duplex/speed and no more errors.

There does seem to be some inconsitency, i.e. I just noticed on one particular workstation, the connection is dropping every 15 minutes, as opposed to the Canon printers dropping every 10 minutes. As at first, we did think it was a Canon printer issue, till we seen it happening to other hosts. Some of the machines also go for a while without dropping, then experience the problem then it appears to go away as such. I need to do abit more investigation, and provide abit more detail. The network consists of approx 80 hosts, and from I can tell about 30% appear to be affected.

No IP phones, on the network. Everything is patched directly through, this customer did have desktop switches out on the floor, but we got rid of these, hence us installing more ports.

Wake on LAN, good call, didnt think of this. Not sure will look into this and provide an update.

Unfortunatley we dont have a smartnet agreement, so unable to open a TAC.

I'll get back with config and further info.

Cheers

Paul

burleyman · ‎08-23-2012

Can you post the config of one of the switches that is having this issue.

Mike

TrustNetworks · ‎08-23-2012

Hi Guys

Sorry for the delay in response, was trying to get a clearer overall picture of what is happening.

It is starting to look like this is host specific.

There are only handful of workstations, that apear to be experiencing a problem, and it apears inconsistent on the workstations. I think these would need to be looked at individualy. Maybe cabling, driver issues, or anything.

With regards to Canon printers, the customer has 8 in total.

2 are currenlty offline, and cant check.

3 are models iR3225 and iR3235, these are dropping consistently every 10 minutes. Although 1 behaved itself this morning.

3 are models C5035, 1 of these was dropping every 12 minutes yesterday morning, but apart form that these seem to be behaving.

In the image below, hosts.jpg the top 2 graphs show a couple of the workstations affected, the bottom 2 graphs are 2 of the iR model printers.

We think with STP currently disbaled this problem is going relatively unoticed.With STP enabled, each of those outages lasts around 90 seconds. So you can imagine from the users perspective the problem appears particualy bad.

Regards

Paul

burleyman · ‎08-23-2012

This is strange...

On one of the printers that is having this issue please give this a try. Disable all protocols that are not needed, i.e AppleTalk, IPX, etc. Next Totally disable SNMP, it is probably on by default. The SNMP did cause some of our printers to stop responding periodicly, not sure why but when I disable SNMP the issue went away. We have now gotten new printers and actually use SNMP on them now and it seems to work on these new printer, so it was a head scratcher.

As for the workstations, could the NIC's be set to going into power save mode or sleep. I would check this for the printers as well, we also had to turn that off during business hours as it did cause issues.

Mike

TrustNetworks · ‎08-23-2012

Hi Mike

Appletalk was off, have disabled SNMP, although I think this might be required for the billing software. Also disabled WSD and Multicast discovery. None of these have had an effect, but I think as we have narrowed it down to a particular models of printers, we can escalate this to the printer company, as it now doesnt appear to be an issue with the switches or the infrastructure. Chances are, the issue existed before we swapped out the switches, but the STP has only highlighted it.

The workstatiosn, should have all power save features disabled through GPO, however we need to investigate this, maybe tcp offloading or some other fancy thing is playing a part here.

Regarding STP, it appears to take around 90 seconds to run before the port changes to the forwarding state. Does this depend on the network topology. This seems to be causing problems with some of the WIndows 7 machines booting up quickly and users logging in with everything offline, is the answer to always have these ports set to portfast?

Cheers

Paul

Leo Laohoo · ‎08-23-2012

It is starting to look like this is host specific.

Sorry Paul, but I ain't trying to pick an argument with you. I just need to understand a few bits and pieces here.

What made you start to suspect it's "host specific"?

There are only handful of workstations, that apear to be experiencing a problem, and it apears inconsistent on the workstations. I think these would need to be looked at individualy. Maybe cabling, driver issues, or anything.

I'm really leaning towards cabling here.

Here's another way of testing, do you have a spare switch? If you do, take this to the an area where there's a significant amount of clients. Run cables to the clients to this spare switch and connect the uplink back to the network.

If the clients' link flap, then this takes away it's a cabling issue. If the uplink drops (Cisco to Cisco link) then that eliminates it's a Cisco-thing and points to the general direction of cabling.

TrustNetworks · ‎08-24-2012

Hi Leo

No worries, more like constuctive conversation, host specific because the printers, and specfic models are exhibiting consistent behaviour. Its unlikley that only those printers would have cabling issues, and the rest of the network being fine, other than a handful of workstations.

Bearing in mind when this case was first escalated to myself, the report was "everything on the network was going down", while working through with you guys and doing my own investigations, ive been able to get a clearer view of the overall picture, and no longer think it is a "Cisco thing", other than in previous post I mentioned above that I believe having STP running has drawn attention to some possible pre-existing issues. It did appear intitially that the printers and workstations appeared to be having the same issue, but now I think they may be isolated faults.

I wouldn't rule out that maybe one or two of the workstations are cabling faults, i.e. a lead has been kicked loose out the floorbox or back of the machine. Your suggestions are very valid, and if I was onsite it would be trivial to try these out and eliminate the problem further. However managing the network remotley we have to work in reverse, and rule anything else out, before incurring the cost of deploying an engineer to site. I guess i was hoping to use the extensive capabilites of the switches to narrow down the issue futher, or hear from anyone that has seen this issue before. Particulary as on the outset, the issue appears identical to this one http://www.experts-exchange.com/Hardware/Networking_Hardware/Switches/Q_27639662.html and I have also witnessed this happening on another of our own customers network, which seemed to disappear by itself without us getting to the bottom of the issue. Searching around the web, it does appear several others have posted similar symptons, but there are few fixes posted.

Cheers

Paul

P.S. not sure where you are, but bank holiday weekend here in the UK, have a good weekend

Leo Laohoo · ‎08-24-2012

P.S. not sure where you are, but bank holiday weekend here in the UK, have a good weekend

I'm here in Australia but I'll be dropping by the UK in the next few weeks for a 5 weeks tour of Europe.

TrustNetworks · ‎08-24-2012

leolaohoo wrote:

P.S. not sure where you are, but bank holiday weekend here in the UK, have a good weekend

I'm here in Australia but I'll be dropping by the UK in the next few weeks for a 5 weeks tour of Europe.

Cool, bring some spare patch leads with you

Leo Laohoo · ‎08-24-2012

Cool, bring some spare patch leads with you

Better not be a joke because I have about three boxes of patch cords I've embargoed from 98 sites.

Dave Rinker · ‎11-07-2012

Paul,

I've got the same problem at my site.....

Every 10 minutes Cannon iR3235 printers bounce. WS-C3560V2-48PS 12.2(50)SE1 ipbase

I just hard coded the printer/port and am debating on shutting off STP on that port but we're stable for 4d09h so I don't believe this will help.

I also have two VLANs on these ports (data/voice) and have stripped the voice and port security off to bring it down to the basics.

No interface errors.... just bouncing... we have several of these iR3235 printers and I'll post back with the results of changing the one as a gauge against the others.

Please post if you find a fix!

thanks,

Dave

TrustNetworks · ‎11-08-2012

Hi Dave

I actually now suspect this is nothing to do with the Cisco equipment. Only that the delay that both STP and the link negotiation creates in bringing up the interface, has highlighted an existing issue with the Canon printers, that previously went undetected.

It does appear when you make changes, sometimes, not always the printers behave themselves for a while, then the problem re-occurs. Perhaps, it is a case that whatever the printers are doing every 10 minutes, always occurs, but when a change is made something is reset and what ever is happening is that quick that it is undetectable. i.e. the switch doesn't detect the interface dropping, but over time the printer slows down, and the drops are detected.

As in our case, I belive the drops are only lasting a few seconds, this is going undeteced by the users and while not fixed, is not causing an issue.

It is however, good to know we aren't alone in seeing this issue. Also let me know if you discover anything, we have spent a significant amount of time investigating this.

Regards

Paul

Dave Rinker · ‎11-12-2012

Paul,

Yeah, no change after removing the voice vlan (which wasn't used just a default of ours) and hardcoding the duplex/speed.

Same bouncing issue.

Nov 12 07:07:22.235 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 07:17:11.893 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 07:17:13.915 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 07:27:18.157 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 07:27:20.162 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 07:37:11.847 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 07:37:13.852 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 07:47:17.945 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 07:47:19.950 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 07:57:11.807 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 07:57:13.820 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 08:00:01.481 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 08:00:03.486 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 08:01:42.789 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 08:01:44.794 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 08:11:49.082 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 08:11:51.087 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

Nov 12 08:16:00.781 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down

Nov 12 08:16:02.786 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to up

!

interface FastEthernet0/17

description Cannon Printer iR3235

switchport access vlan 13

switchport mode access

speed 100

duplex full

spanning-tree portfast

spanning-tree bpduguard enable

end

If I figure this out I'll be sure to update this thread.

Thanks,

Dave