IE4010 10Mbit half duplex issues

[andreas.dahlberg]

Hello everybody! 

Update! A bug has been reported; https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvo93073

We have just started our deployment of IE4010 switches in an industrial environment where we are replacing old 2960S switches.

However just after we had replaced about four switches the OT guys started reporting issues with the connected PLCs. The SCADA system was reporting connection issues and one of the controllers even stalled because of this (they say it is a feature..?). 

The problem seems to be with the controllers connected by 10Mbit Half Duplex. 

Configuration is exactly the same on the new IE4010 switches as the old ones.

We usually static configure 10Mbit Half duplex on these ports as it is recommended from the PLC vendor.

Here is the interface configuration.

interface GigabitEthernet1/0/1
 description PS80_152 CN80
 switchport access vlan 180
 switchport mode access
 speed 10
 duplex half

Interface Input and CRC error counters are increasing on the IE4010 switch. 

This is from one of the IE4010 switches still in production:

GigabitEthernet1/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 00bc.60e2.b401 (bia 00bc.60e2.b401)
  Description: PS84_178
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 254/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex, 10Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 10000 bits/sec, 13 packets/sec
  5 minute output rate 34000 bits/sec, 30 packets/sec
     18178818 packets input, 2035872963 bytes, 0 no buffer
     Received 1144670 broadcasts (1144652 multicasts)
     47268 runts, 0 giants, 0 throttles
     67477 input errors, 18063 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1144652 multicast, 0 pause input
     0 input packets with dribble condition detected
     35866394 packets output, 5726349001 bytes, 0 underruns
     0 output errors, 30178 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 44909 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

This PLC connected is not doing any serious applications, just controlling a pump.

We have tried turning off PoE on the IE4010 ports and using auto speed and duplex settings.

Haven't found any experiencing the same problems, nothing in the bugtracker on this either...

Anyone with any ideas? We have started a case with our VAR, haven't heard from them yet..

[Mark Malone]

Hi
did you try hardcode 10mb full duplex just to see or even a 100 half , lot of errors there which can arise from half duplex setup in general but if the vendor says there half dup it should work

have you checked the cable long shot as 4 effected but just incase run a TDR test see if there ok

http://packetlife.net/blog/2008/mar/28/investigating-ciscos-built-in-tdr/

id clear the counters too to get a real picture of what errors are coming in real time , clear counters gig 1/1

[andreas.dahlberg]

The PLCs in question has only 10Mbit half duplex interfaces.

With my PC connected, i did check with 10Mbit Full duplex and that is ok. 

We have tested several cables, and when i'm now troubleshooting with my computer, i have racked up a IE4010 in our lab. 

Testing 100Mbit Half duplex is on my list to check if that is also an issue.

A TDR show the same result on both IE4010 and a 2960S switch connected to the same PLC:

2960S:

TDR test last run on: February 08 11:09:11

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/1   10M   Pair A     6    +/- 0  meters N/A         Normal
                Pair B     6    +/- 0  meters N/A         Normal
                Pair C     7    +/- 1  meters N/A         Short
                Pair D     6    +/- 1  meters N/A         Short

IE4010: 

TDR test last run on: February 08 11:02:07

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/1     10M   Pair A     7    +/- 0  meters Pair B      Normal
                Pair B     8    +/- 0  meters Pair A      Normal
                Pair C     8    +/- 1  meters Pair C      Short
                Pair D     8    +/- 1  meters Pair D      Short

[andreas.dahlberg]

A Case is opened with Cisco TAC and they are digging into it. We have provided packet-captures and switch IOS "show" information a week ago. No update from TAC yet.. :/

[Richard Burts]

I am noticing the number of collisions and of late collisions. This is sometimes evidence of duplex mismatch. 

HTH

Rick

[andreas.dahlberg]

Hi!
Yes, that is what we though to, but the devices we are connecting has a static 10Mbit/Half duplex interface so the switch port is correct. And it is not the collissions that are the problem, they are expected. it is all the CRC and Input Errors that should not be there.. :/

[wesleyribeiro]

I had the same problem but it was with the Cisco Catalyst IE3300 line Manually Configured the Port - Automatic - Verified Fiber and UTP cables Due to urgency I changed to the IE3000 models that are working without CRC error

Details I noticed:

IE3300 - GigabitEthernet Ports

IE3000- FastEthernet Ports

[pr3951 Pedro Riley-Olvera]

I am having same issue with 10m half duplex moving from 2960s and C3750 to C3850s and C9300s. Yes, the collisions are expected and noticed. Also seeing Xmit errors on the counters but with store-and-forward as default those should be getting dropped. Only thing that works so far is to put a dumb switch in-line. This stops user end comm alarms. We tried putting the old 2960 in between with one port in and one port out same VLAN and also we dont see errors. We tired replacing end equipment, cabling and different ports, and hard coding 10m/half duplex but nothing works except the dumb switch in between or the 2960 in between. All are not reasonable solutions. 

[Joseph W. Doherty]

I believe the underlying problem is, network equipment vendors have "decided" 10 Mbps half-duplex Ethernet is so "old", no one still uses it or really needs it.

[adsyparker]

For others dealing with these issues, we have discovered that our issues with half duplex connections were port specific.

It seems the first port in each group was suffering with late collisions.  This must be some hardware flaw with the 4010.

Gi1/1, Gi1/7 and we assume the same for Gi1/13 and Gi1/19 but did not test.

Once we were satisfied that it worked on Gi1/2 we moved over and problems went away.

As a note this is very firmware specific, we had lots of problems with links completely dying on half duplex connections with newer firmwares.  We are currently on 15.2(4)EC2 with a IE4010 and it is stable and has been for some time.

[Leo Laohoo]

Our standard version for the IE 4010 is 15.2(7)E5 and we have not seen this behaviour before.