network loss please help me as soon as possible

[mauleshg1]

We have cisco 2960 switches and cisco 3750 l3 switch,this switches connected in stack 

now we have scada machine more than 60 (PLC) in our plant and and it is connected with cisco switch.

Now  we are facing problem is that after certain time of period machine (fix two machine) lost the network connection and it gives emergency so after emergency machine stop the production we have to reboot it and when we check the ping status at the time of emergency, we are getting ping fine no any break or request time out.

now customer is saying that this related to network problem like network load, network loss so that our machine gives emergency.

now how can i troubleshoot the problem customer told us that they have already talked with

machine vendor and that vendor came and check machine is ok and in that scada machine allen bradley switch is installed..

now help me as soon as possible it is very serious issue...

[Charles Hill]

If the machines appear to lose connectivity or stop working, but is still responding to pings, possible duplicate ip address?

I would verify the arp table for the ip address of the machine when its working and then verify the arp table when it stops working. 

Verify machine's mac address is still binded to the machine's ip address.

show mac address-table interface g0/1 ( to obtain the mac address of the machine)

sh arp | i xxxx.xxxx.xxxx (mac address of machine)

will return ip address and mac address of machine

re-type the commands above when machines stops working to verify mac address is still binded to the ip address.

Seen a scenario before where pc's would randomly lose connectivity.  Issue was an end user connected a linksys router to their network with the same ip address of their gateway.  PC's would bind their gateway address 192.168.1.1 to the mac of the linksys, which would result in no access to their gateway. 

To locate a mac address type

show mac address-table address xxxx.xxxx.xxxx

If the machine tests ok, I would do the above test again on the ip address the machine is trying to communicate with

This is a start.. 

[mauleshg1]

I tried sh arp | i xxxx.xxxx.xxxx (mac address of machine) but it is not showing ip of machine.

and i tried show mac address-table address XXXX.XXXX.XXXX

SHOW WHAT IS NEXT..PLEASE HELP ME

[Charles Hill]

I assume its down now...

Ping the machine and verify if its responding.

If it does respond, disconnect the ethernet cable going to the machine and verify if the pings drop.

You can do a continued ping by typing

ping 10.1.1.1 -t

[mauleshg1]

Machine is not I am getting ping response So it is not down

[Charles Hill]

I would disconnect the ethernet cable going to the machine and verify the pings stop.

[mauleshg1]

That I have checked

[Charles Hill]

Jon asked a great question, Are all other machines ok?

All on the same network?

As jon, pointed out, if this is an urgent issue, you would probably be better opening a tac case, and they can assist you.

If all other machines are ok and its all on the same network, and the machine has connectivity, it sounds like its something to do with the machine.

But its hard to say.

[mauleshg1]

All other machine is working OK is there any way to check the traffic of that machine Or any command which help me.

[Jon Marshall]

Have a look at interfaces on the switch that connect to the machines and see if there are errors, collisions etc.

Check the configuration of the ports compared to ports that are working okay.

Is there any difference, are they in the same vlan etc.

You need to narrow down the issue by eliminating things.

Jon

[mauleshg1]

that we have checked port configuration is same as other ports  and how can i see there is collision and error on that port or not is there any command or what.

[Charles Hill]

show interface g0/1 will show you crc errors, collisions, etc...

Look for speed and duplex status as well.

Preferably both sides will be configured with auto-negotiation. 

Regardless, both sides need to be configured the same.

Auto - Auto

full 1000 - full 1000

full 100 - full 100

[Charles Hill]

You can do a packet capture to see whats going on.

you'll need to download wireshark, if you don't have it already.

Here is a sample config for a monitor session on a 3750, which will allow you to capture the data on that port.

Switch(config)# no monitor session 1

Switch(config)# monitor session 1 source interface gigabitethernet1/0/1

Switch(config)# monitor session 1 destination interface gigabitethernet1/0/2 

Connect your laptop on the destination port and launch wireshark and capture the data.

[mauleshg1]

after capture the data what i have see in capture data can you tell me

and our scad machine manufacturer is rockwell automation and inside rockwell automation plc allen bradley switch is installed and from that switch network card is connected and allen bradley switch is connected with 2960.

topology is as below.

network card(PLC) - 2960 switch - 3750 switch - server..

[Charles Hill]

Not sure if this would be any help, but here a link to a forum discussing troubleshooting scada machines.

http://forums.ge-ip.com/printthread.php?t=9319