Troubleshooting network with poor backbone

[mosaster]

Hello folks,

I'm working as a network engineer for small industrial equipment manufacturer. I already have Cisco entry certificate, been successfully applying Cisco knowledge in my work. However, this time I got a broken and malfunctioning network that caught me clueless what to put my hands to first. I expect there would be more such cases soon, so I decided to ask other specialists here how to troubleshoot poorly designed networks.

The network has some serious structural weakness:

- it is a purely wired network based on several managed switches along with unmanaged 'dumb' switches. There are only 20 machines being once in a while towed and plugged in to a different socket. Bad news there are waaay more dumb switches than the machines. Dumb switches are network sockets to the machines, but also interconnected and join different branches together(!) in the same VLAN(!). Since dumb switches provide no 'under the hood' info there is no way to find where a loop may occur. Connections between nodes are good deal tangled too. No topology available as well as no meaningful hierarchy between the nodes.

Network equipment comes from various manufacturers adding complexity to the task. Good thing I know how to configure it. Thus the network came this odd and unstable making them call me.

Went there yesterday to perform initial survey. What I found so far:

- the network is purely mesh, no core

- the network has no routing whatsoever, everything is within the same VLAN. This is not the best strategy for sure, but I would accept it since there is no meaningful structure anyway, the cables are already in the ground.

- no trunks

- there is a main computer polling machines for operational data

- RSTP enabled, with no BPDU guard available, I assume there is a tree, but supposedly unstable (god knows why), since...

- drew up comprehensive ping statistics, there is packet loss to some machines as well as switches, with unstable ping times to them

- compared ping straight to machine port and network device, loss occurs only within the network

So, folks, what would be your next move?

Regards,

Alex

[balaji.bandi]

Good Stuff - Here is a suggestion to improve network stability. - to do that you need to start below steps (my point of view)

1. Draw a small network diagram, which shows what port connected what device.

2. List what Device models you have Cisco and another brand switch - how they connected.

3. Did they all in the Same Area within Limit of Cat 5 / Cat 6 range distance?

4. if so ask business why they connected (dumb switches - i belive this was grown organically) - no time to replace due to business need (they have to keep extending the network without thinking logically and technically, it happened manufacture unit, since their main focus manufacturing(nowadays everything it IT - we need to keep that in mind)

 - other note i have worked some PLC with manufacture units for short period they are not much-concerned network and security, anyone can login to PLC (Scada based) - and do anything, they did not realize - one change can bring the whole plant of manufacturing units.

5. For Improvement - make your own style and like to improve the network.

6. Post Managed switch config along with version running. - Ask Management is this possible to replace dumb hub/switch with Managed switch for better performance with some cost involved to improve the services.

7. keep do your diagnosis from L1 to L3 - (like cat5 cable replacement, making different broadcast domain, with different VLAN possible) - since only 20 Device, its good to be 1 VLAN, if possible make different department different VLAN, so you can have ACL for Security reason.

This is the start. many things can be done...but make sure you fix the Ping Loss internal network so all working stable.

Then ask for IT Funds for better security and other features to add on.

Is this Make Sense?

[mosaster]

Yes, all recommendations absolutely make sense.

Diagram is definitely the first thing to do, although it would take me probably couple of days :)

Yep, the machines are based on PLC controllers using ISO TCP therefore re-transmissions and sessions drops are their weak point.

@balaji.bandigenerally agree with your thoughts on the matter :)

[balaji.bandi]

if the PLC not able to get right information from main Server / Control. Your plant may not run or get optimal results, Mechanical / Electrical / Automobile engineers have no clue what is going one.

So discuss with Management and upgrade the basic switching system to meet the business need. So business can run smoothly.

had the client has the same issue, big outage, due to improper SCADA deployment with PLC. (they think any local available hub can do the job, and they spent millions on SCADA)

Once the re-designed network and replaced decent switches, those days (3750E) were the latest, all good and still running as expected more than years.

[Leo Laohoo]

Draw the network.