Contents
- Authors
- Executive Summary
- Operational Technology (OT)
- Industrial Automation and Control Requirements
- Security
- Network Resiliency and Uptime
- Environmental
- Connectivity
- Upgradeability
- Resources and Access to Networking Skills
- Network Segmentation
- IP Addressing
- Era of Software-Defined Networks
- Enterprise IT/OT Non-Fabric Design
- Enterprise IT with Partial OT SD-Access Fabric Design
- Enterprise IT/OT SD-Access Fabric Design
- Shared Enterprise IT/OT SD-Access Fabric Design
- Deployment Considerations
- Summary
- References
Authors
Mahesh Nagireddy
Technical Marketing Engineer, Technical Leader
Keith Baldwin
Senior Technical Solutions Architect
Paul Didier
Solution Manager
Executive Summary
Digital transformation creates new opportunities in every industry, and manufacturing is no exception. When designing Operational Networks, we based our designs on the Purdue Model where Layer 2 (Access) networks are at the edge supporting Industrial Control applications and Layer 3 (Core and Distribution) connects them to Plant-level applications. However, significant changes have occurred in the networking industry over the last 30 years, especially around automation and flexibility referred to as Software-Defined Networking. Manufacturers want to reduce cost and downtime, improve turn-around times, adapt to new regulations and compliance issues, and deliver better service for increasing customer demands. Along with these challenges, manufacturers need to address post-pandemic difficulties with supply chain delays and position themselves to outgain competitors to benefit from tremendous market growth opportunities. Software Defined Networking is well-positioned to drive the Digital Transformation and provide Manufacturers with key improvements.
Today’s manufacturing leaders focus on adopting more reliable factory automation tools and tighter data security models and doing more with fewer resources. In this revolutionary landscape, more machines are connected to the network, outfitted with sensors enabling real-time communication with other devices and human operators. With this rise in connected machines, there is a need to automate and segment, providing the right level of access.
As the network becomes more critical, OT Personnel still need to rely on the network to operate the production environment, often without tools that tell them how the network is performing, applying consistent network configurations and ability to maintain and update the network after initial deployment. Hence OT personnel having different access can still benefit from the new network and security management tools used by IT.
This whitepaper provides design guidance for the factories and production facilities focusing on utilizing the innovations in Cisco DNA Center and SD-Access to create simple, secure, and flexible production networks for Operational Technology (OT) and how Cisco DNA Center and Cisco SD-Access help implement and integrate OT and IT Networks.
Operational Technology (OT)
The Manufacturing ecosystem is rapidly adopting Cloud and Edge applications for mobility, flexibility and faster response to changing demands. The rapid digitization of OT has resulted in connected control systems, growth in the use of IoT devices, and the need to derive insights from collected operational data. Enterprises want to innovate and differentiate their offerings beyond the traditional carpeted spaces. In addition to extending beyond traditional air-conditioned areas, there is a requirement for a consistent network policy to address security concerns for extended networks.
IT and OT are often two different departments within a single organization. An OT network is typically a separate physical network from an IT corporate network and usually supports a direct line of business. OT networks are described as a spectrum of technologies, including traffic types, flows, topology mandates/restrictions, segmentation, Etc. These components can be introduced if the organization is part of a regulated industry or if there are existing norms within specific sectors that have lived for many years.
Other unique requirements include lossless LAN duplication, Supervisory Control, and Data Acquisition (SCADA) protocols, and having elements in sync to a sub-microsecond accuracy. Manufacturing includes Industrial Automation with Profinet protocol and a strongly segmented hierarchical approach, usually in line with IEC 62443.
OT networks often have their own IT infrastructure, and, depending on the size of the OT department, staff usually have varying networking skill sets. Firewalls are used to separate the IT/OT environments due to security or compliance requirements and have several benefits like Policy enforcement, User Control, Threat Containment, and Change Control. This firewall separation ensures that an event, such as an accidental change on the IT or OT network, should not impact other networks.
Challenges with OT
Industrial Automation and Control Requirements
The core of modern production facilities are the Industrial Automation and Control Systems (IACS). IACS Applications and IACS devices are important for smooth plant operations and functioning. These systems are made up of devices such as a PLC, sensors, actuators, controllers, machine tools, drives, robots, HMIs, industrial PCs (IPCs), and any other IACS devices that need to communicate in real-time. More forward-looking industrial automation customers are looking into technology changes that allow them to virtualize Programmable Logical Controllers (PLC), which will gain them a better level of agility. These systems operate in a section of the facility called the Industrial Zone and perform a function referred to as a Cell/Area Zone. The devices communicate with each other to maintain the operational integrity of the production – to keep the plant operational. The frequency of the communication is measured in milliseconds or even 100’s of microseconds and systems are very sensitive to packet loss, latency, and jitter. The systems use distinctly different communication protocols (e.g., Profinet, the Common Industrial Protocol or CIP, Modbus, etc.) from IT, although many of the IT protocols are also found in these systems. More stringent applications, such as high-speed motion, often require tighter synchronization and need precise time services from the network (e.g., Precision Time Protocol (PTP)). These systems are often designed, deployed, and operated by an ecosystem that does not have IT, Networking or Security expertise. And the systems scale from thousands of devices to tens of thousands in a single production system. In summary, the network design and configuration need significant accommodation to support the IACS.
Security
Protecting IACS systems from cyber threats is top of mind and these systems have evolved over time. The core IACS often have an implicit trust of other devices on the network. As the ecosystem has adopted Ethernet and open networking standards over the last 20 years, so have the cyber-security challenges increased. But even with the adoption of these technologies, the systems remained separated (a.k.a. air-gapped) often with architectures with many unconnected islands of networks. As Industry 4.0 standards and Digitization have driven manufacturers to connect these systems and adding cybersecurity as an afterthought. As a result, most OT networks use a firewall and De-Militarized Zone (DMZ) concept to secure communication between the production systems and other Enterprise or external communications. As they are adopting more cloud-based applications and connectivity, the firewall-based DMZ is no longer sufficient as a security model, requiring more security visibility, segmentation, and protection within the production systems themselves.
Network Resiliency and Uptime
As OT networks have become highly critical for the functioning of an organization, and in many cases, an OT network is the core of a company’s standing. It is vital to provide resiliency to full redundancy options for such networks. Thus, if the network is impacted, chances are incredibly high that the plant or parts of the plant will be down, potentially costing millions of dollars per minute or hour. As a result, a host of topologies and Layer 2 resiliency protocols are used so that if a failure occurs communication continues, and the plant stays operational.
Environmental
OT networks can be indoors (uncarpeted space) or outdoors or both. In many cases, both indoor and outdoor models will not be temperature-controlled and will have to withstand harsh conditions exposing equipment to a significant range of mechanical, chemical, temperature, and electrical challenges. The network equipment must be ruggedized, or operate in ruggedized housing, with significantly better Mean time between failure (MTBF) than IT equipment due to the criticality of the systems that rely on it.
Connectivity
Linear Daisy-Chain, Star, and Ring topologies might have to be utilized with fiber resources available. It is also common in OT networks to have elements far more geographically distributed compared to IT networks. Due to these geographically dispersed installations, it becomes impractical or too costly to “home run” every access switch back to the Distribution Layer.
Upgradeability
Uptime is critical on the manufacturing floor, especially within industrial automation and utilities. Production facilities have occasional maintenance windows where changes can be performed. During these maintenance windows, network services are often imperative, making it very challenging to upgrade the network itself. The systems and networks in OT environments are not updated anywhere near the same frequency as typical IT networks which creates significant even more security challenges.
OT has a unique requirement for increased efficiency during deployment and upgrade of the network and endpoints, such that the OT team can add new network hardware easily and RMA network hardware, when required, with minimal business disruption.
Resources and Access to Networking Skills
Operational environments are generally the responsibility of the Operations organization, in other words, the COO. These environments are designed, deployed, and operated by teams with specific manufacturing skills and capabilities, often without networking and cybersecurity knowledge, much less skills and capabilities. This includes the networking infrastructure used in the production facilities. And, due to that lack of “networking” background, many IT organizations do not allow access to network management tools to the Operational teams concerned that such access could result in unwanted or unexpected actions or inappropriate access to other systems. Although operations rely more and more on networking and cyber security technologies to keep the factory running, the operators lack the tools and skills to operate and maintain those technologies.
Network Segmentation
Today’s OT network occasionally provides limited segmentation capabilities, and mostly these networks are VLAN-based. If IP ACLs instantiate segmentation, they can become difficult to scale, troubleshoot, and maintain over time. When deeper segmentation is needed, many organizations create physically separate OT networks. There is a need to solve this by providing logical segmentation with the strength of physical separation but with the simplicity of a single network.
IP Addressing
Whereas in most IT networks, the devices get their IP address dynamically when they access the network. In many industrial networks, devices are assigned static IP addresses. The industrial applications are programmed with specific IP addresses vs. using naming services. And, in some cases, these applications are replicated across Cells or Areas so that many devices may have the same IP address. To provide connectivity beyond the Cell or Area, network services may be required to translate network addresses (NAT), either an L3 or an L2 NAT. This provides challenges to management and telemetry applications that may be on the other side of the NAT boundary.
Era of Software-Defined Networks
Network management in the IT ecosystem has gone through a “Software-Defined” evolution to help IT organizations deploy, operate, and maintain networks with fewer resources, less operational efforts, more automation, and improved uptime. Cisco DNA Center is the core SDN orchestration platform for the automation of wired and wireless networks and also adds the benefits of machine learning to provide assurance that the network is delivering the business intent.
The business and technical requirements of OT networks, agility, harsh environments, segmentation, etc., can be met by introducing the Cisco SD-Access Fabric Solution, which is a network architecture orchestrated using Cisco DNA Center. Cisco SD-Access solution gives OT a curated view and set of functions to perform key network tasks consistently and in a scalable manner, where Cisco ISE is also an integral part of the overall solution. The SD-Access architecture comprises a fabric technology with programmable overlays and easy-to-deploy network virtualization, which enables virtual networks (overlay networks) delivered over a physical network (underlay network) to meet the design intent. In addition to network virtualization, fabric technology in the campus network enhances communications control, providing software-defined segmentation and policy enforcement based on user identity and group membership.
OT can no longer be secured only by firewalls primarily designed to separate IT and OT. Innovative approaches are needed to leverage security models developed by IT, such as Zero-Trust, which rely on deeper visibility into connected endpoints, their role in operations, and their interaction with other endpoints, enabling through visibility the development of access policies and network segmentation.
The table below describes how Cisco DNA Center and Cisco SD-Access Fabric help resolve the Manufacturing challenges described in the previous section.
| Challenge | How it is met |
|---|---|
| Security | Cisco SD-Access and Identity Service Engine improves the security posture of the network by shrinking the attack surface and employing Zero-Trust principles for Network access. |
| Network Resiliency and Uptime | Cisco DNA Center can monitor the network infrastructure, automatically identify issues, and recommend resolutions to improve uptime. |
| Environmental | Cisco’s Industrial IoT product portfolio is designed to work in most industrial environments and is deployed, configured, and managed by Cisco DNA Center. |
| Upgradeability | Cisco DNA Center is designed to automate the upgrade of network infrastructure devices at scale using SWIM (Software Image Management) features and enforce consistency of software versions across the network and help ensure compliance. |
| Resources and access to Network skills | Cisco DNA Center and SD-Access fabric encapsulate IT network expertise and automatically deploy that via scripts and workflows, reducing the amount of work on Network experts |
| Network Segmentation | Cisco SD-Access automatically applies and enforces network security policy that includes both virtual network at the macro-segmentation level and security group tags at the micro-segmentation level. |
| IACS | Industrial applications require resilient, flexible secure networks. DNA-Center and SD-Access help deploy and manage the industrial network to deliver and ensure these requirements are met. |
The following sections will outline how Cisco DNA Center, Cisco SD-Access and Cisco ISE solutions can meet these business and technical requirements. Each solution will be based on real customer OT environments to enhance OT network performance, availability, security, segmentation, and increase network agility.
Enterprise IT/OT Non-Fabric Design
Alongside SD-Access Fabric deployment for Enterprise IT, the solution described here applies the usage of Cisco DNA Center for Non-Fabric OT Environments which gives OT a curated view to perform critical network tasks. Those critical network tasks include basic network automation, assurance and monitoring of the production network, guided remediation of identified problems, and replacement of Fault devices.
Figure 2 below provides a logical view of OT architecture that needs to be customized for specific customer applications and connectivity requirements.
Figure 2: Cisco DNA Center Managed Non-Fabric IT/OT Environment
It is recommended to add a Cisco DNA Center as an application in the Industrial site Operations zone to automate specific functions and monitor the operational status of the production environment. A DNA Center instance that supports both Enterprise IT and Production OT networks may lead to inadvertent changes or updates impacting the production system that could lead to downtime. A separate instance of Cisco DNA Center for OT environment helps ensure operational requirements are maintained. This design also simplifies performance requirements and keeps communication flows between Cisco DNA Center and Cell/Area zones switches in one network, making it easier to meet design recommendations.
Monitoring and Managing Cisco Catalyst and Cisco IE switches in OT Environment with Role-based Access control and Integrating Cisco DNA Center with Cisco ISE and Cisco Cyber Vision provides secure access to the network based on the Zero-Trust Model.
Enterprise IT with Partial OT SD-Access Fabric Design
This design has a single Instance of Cisco DNA Center in the Enterprise zone with two Fabric Site Architectures. Along with the Enterprise IT Fabric Site, the OT network has a dedicated Cisco SD-Access OT Fabric Site not extending beyond the Industrial site Operation Zone. Having separate Fabric Sites for IT and OT might be an organizational decision driven by security policies, compliance, or departmental boundaries. This design provides complete physical separation between IT and OT networks while maintaining common segmentation constructs such as VN and SGT. It also reduces the risk of impact when changes are implemented on either side.
Figure 3: Enterprise IT with Partial OT SD-Access Fabric Design
Another advantage of this design is that the Enterprise Cisco SD-Access Fabric Site and dedicated OT Cisco SD-Access Fabric site share common components such as Cisco DNA Center and ISE. With the future Cisco DNA Center releases supporting site-based RBAC, this design provides a true separation from the logical perspective. Still, it does share the physical Cisco DNA Center appliance/cluster between IT and OT Cisco SD-Access sites. Cisco Cyber Vision is a valuable tool in OT networks, which helps to identify and classify OT devices in conjunction with Cisco ISE.
The OT Fabric Site is separated from the Enterprise Cisco SD-Access Fabric Site with a firewall. The firewall has at least two zones, one connecting to the OT Fabric Site and another to the Enterprise Cisco SD-Access Fabric Site. The firewall can be a Layer 2 (transparent) firewall or a Layer 3 (routed) firewall. At an intersection, we can allow OT devices to communicate at Layer 2, with very low latency, without hairpinning the traffic via a Peer (Fusion) router/firewall. If needed, the firewall could NAT the traffic from the OT site toward the Enterprise Cisco SD-Access site. Using SGTs, we can further enforce micro-segmentation since Cisco SD-Access gives us a highly automated way to deploy a micro-segmentation policy.
Notably, the Border Nodes and Control Plane Nodes in the OT Fabric reside in air-conditioned spaces similar to Enterprise IT Switches and routers. The network is extended to the non-carpeted areas by connecting industrial switches to the distribution layer. Along with Edge Node role functionality on distribution layer switches, redundancy protocols like Stackwise or Stackwise-Virtual should be used for High Availability. To choose the appropriate Border, Control plane, and Edge node switches, please refer to Enterprise Product Datasheets or the Cisco SD-Access Compatibility Matrix.
A cell Area/Zone is a functional area within a plant facility, and many plants have multiple Cell/Area Zones. This is typically a non-carpeted space of the OT network. A Cell/Area Zone has a wide variety of devices like HMIs and Control room workstations for runtime supervision and operations, Controllers to direct and manipulate the manufacturing process, and a Wide variety of Sensors and Actuators for basic industrial processes. All devices in the Cell Area/Zone are connected to the Cisco IE family of switches which are purpose-built for operating conditions.
This design can also support two separate instances of Cisco DNA Center, one for IT in the Enterprise Zone and another for OT Network hosted in the Industrial site Operations zone. Also, to avoid any unintentional changes or updates impacting the production system, Cell Area/Zone devices can either be managed by a dedicated OT Cisco DNA Center or can be unmanaged devices. Most networks in this area carry both non-critical and time-sensitive critical control traffic. With this design, extremely time-sensitive/critical control traffic like Profinet, Ethernet/IP, CC-Link IE, etc.., can be kept outside the fabric as an External Layer 2 switching domain or local to the Cell Area/Zone. For traffic destined outside the Cell Area/Zone will be handled by the Fabric device in OT Fabric.
Enterprise IT/OT SD-Access Fabric Design
This design maps to Industrial Automation Reference Architecture, which describes the core network and security features and functions overlaid on the industry model of Industrial Automation and Control Systems (IACS). This design separates the OT Fabric into two Fabric Sites, the first for the Industrial Demilitarized Zone (DMZ) and the second for the Industrial Security Zone. The OT Fabric Sites are connected to the Peer firewall with multiple firewall zones, one for each OT Fabric Site and another relating to the Enterprise SD-Access Fabric. The OT Fabric Sites bring all the traffic toward the Peer (Fusion) firewall device. The Peer firewall can be a Layer 2 transparent firewall or a Layer 3 routed firewall.
This model maintains separation between the Purdue levels while saving costs using a common multi-zone firewall between the different layers versus dedicated firewalls. This architecture also helps maintain separate macro-segmentation and micro-segmentation within the layers while allowing firewalls to inspect any cross-layer traffic. Within the Cell/Area Zone, if a Broadcast type environment is required for Robots, Measuring Instruments, or Certain PLCs to communicate, we can leverage Layer 2 flooding functionality or even WOL features for silent hosts. Separating the DMZ and Industrial Security zones will limit the broadcast traffic within the cells to stay with the Industrial Security zone Fabric Sites.
This model uses a common Cisco DNA Center for OT and IT SD-Access Fabric sites, but dedicated ISE PSNs are used for OT and IT Fabric sites. Each Fabric site hands off its Virtual network traffic to Firewalls, and based on the Firewall rules, the firewall allows/denies communications. The firewall also provides shared services access to the data center and other services, including DHCP, DNS, Internet, etc.
Careful thought should be employed when deploying firewalls in and between the various zones. The Peer firewall could NAT the traffic from the OT sites towards the Enterprise SD-Access site, as required, but Layer 2 NAT is often done at the edge of the Cell/Area Zone. Enabling features such as inspection and others may have a performance/latency impact, so you should consider how enabling these features could impact an Industrial Automation network.
Figure 4: Enterprise IT/OT SD-Access Fabric Design
Although not part of Purdue reference models, the Manufacturing/Industrial automation solution includes a DMZ between the Industrial and Enterprise zones. Industrial DMZ not only isolates the factory from the outside world but also from its Enterprise networks. One of the critical requirements of the Industrial DMZ (IDMZ) is to secure data transport so that no traffic flows directly between Enterprise and Industrial zones. Firewall rules are employed to ensure Enterprise Zone traffic can reach the IDMZ and the IDMZ can be used to access the Industrial Zone, and vice-versa. The IDMZ allows specific and limited access into the Industrial Site Operations Zone and the Cell/Area Zone. Access is extremely limited, locked down, and audited, with SGTs applied and enforced by Cisco SD-Access becoming an effective method to support these security requirements.
The Cell/Area Zone Process (Level 0) and Control (Level 1,2) is a functional area within a plant facility, and many plants have multiple Cell/Area Zones. Managed and Unmanaged switches do exist in this machine area networks. This design leverages IE switches as extended nodes (EX) and policy extended nodes (PEN) connected directly to Edge Nodes (EN) to extend connectivity and policy to harsh environments and non-carpeted spaces. A Network advantage license is needed to configure the device as PEN. The EN and PEN operate in Layer-2 switch mode and connect back to EN using 802.1x trunk Port channel and are onboarded with zero-touch Plug and Play.
SD-Access can be considered a toolset to simply the network and automate secure segmentation for Industrial Automation. A ring topology for IE switches can be automated in Cisco DNA Center, converting an RSTP ring to Resilient Ethernet Protocol (REP). This change typically results in an approximately 50% improvement in Layer 2 recovery time in the event of a failure. Segmentation delivered on demand where OTs are empowered to create, update, and deploy virtual networks is a real differentiator for SD-Access versus traditional LAN options. This control is available via Rest-APIs, meaning it can be fully integrated into the OT line-of-business processes and applications, truly executing on business intent.
This design also provides an option to carve out multiple OT Industrial fabric sites which could be different sections of the same physical factory or different factories spread across geography. Separating into multiple Industrial Fabric sites affords more isolation and better site survivability. Fabric Site N is unaffected if the Fabric at Site 1 is somehow impacted through operator error misconfiguration.
Connect with your Cisco Sales Representative or Channel Partner to open a Design review request with the Cisco Business Unit for a detailed design review prior to this design and deployment.
Shared Enterprise IT/OT SD-Access Fabric Design
Figure 5: Shared IT/OT SD-Access Fabric Design - Logical View
This simplistic design option enables the use of different virtual networks for IT and OT networks running on a single physical network. Any communication required between IT and OT Virtual networks will be sent through a Peer(fusion) device for inspection, which serves as a common Enterprise firewall. Based on the Firewall policy, traffic can be dropped, permitted, or selectively forwarded. In addition, the firewall also provides shared services access to the data center and other services, including DHCP, DNS, Internet, etc.
Figure 6 below depicts the physical topology of the design where both IT and OT devices are connected to Access ports of Edge Nodes or Extended Nodes, which are mapped to respective Virtual Networks.
Figure 6: Shared IT/OT SD-Access Fabric Design - Physical View
SD-Access support for extended nodes and policy extended nodes is about extending the enterprise network to provide more connectivity to non-carpeted spaces of an OT network. The Cisco Industrial Ethernet (IE) family of switches address harsh environmental conditions, which are purpose-built for demanding operating conditions and can be mounted on 19” rack and DIN-rail form factors.
IT and OT share a physical network but are logically separated and segmented. It is important to understand that while this achieves consolidation, from an OT perspective, additional caveats may require you to take a different approach for some services. In OT, functions that rely on shared services that could take down the environment, whether in this area or across the WAN, should be reasons to pause and consider alternative approaches. Cisco SD-Access is an effective solution because an entirely parallel OT network is unnecessary.
Connect with your Cisco Sales Representative or Channel Partner to open a Design review request with Cisco Business Unit for a detailed design review prior to this design and deployment.
Deployment Considerations
A Fabric Site is an independent fabric area with a unique set of network devices control plane, border, edge, wireless controller, and ISE PSN. The factors of survivability, high availability, endpoint/device count, services, and geography are all factors that may drive the need for Very Small/Small/Medium/Large Fabric Sites.
The Fabric in a Box feature can be used for small and very small Fabric site reference models. The central component of this design is a switch stack or Stack Wise Virtual operating in all three fabric roles: control plane node, border node, and edge node. For Fabric in a Box deployments, SD-Access Embedded Wireless provides site-local WLC functionality. The site may contain an ISE PSN depending on the WAN/Internet circuit, latency, and requirements of the OT Cell.
Extended nodes are switches that run in pure layer 2 mode that do not natively support fabric technology. These Layer 2 Ruggedized Cisco Industrial Ethernet (IE)switches are connected to fabric edge nodes as Extended or Policy Extended Nodes (Cisco TrustSec capable switches) and thus extend the network to the non-carpeted spaces.
A Ring topology for IE switches can be automated in the Cisco DNA Center, converting an RSTP ring to Resilient Ethernet Protocol (REP). This typically results in an approximately 50% improvement in Layer 2 recovery time in the event of a failure. Also, you can see that it is possible to spur off a ring with a linear daisy chain. Current Cisco DNA Center, shipping release support, Extended or Policy Extended nodes up to 18 switches in a single REP Ring.
Cisco Cyber Vision is a useful tool in OT networks to maintain system integrity and production continuity. Cyber Vision helps to identify and classify OT devices in conjunction with Cisco ISE. Edge Sensors installed and running on the Extended Nodes and Policy Extended Nodes capture traffic, decode protocols using the Cisco DPI Engine, and export meaningful data to Cisco Cyber Vision Center. Cisco Cyber Vision Center gathers data and acts as a monitoring, detection, and Management Platform, with Global Center providing alerting and reporting functions.
Summary
Providing automation and segmentation to the Industrial Networks greatly simplifies OT initiatives, accelerating the significant business benefits of digitization. SD-Access alleviates key problems associated with managing and operating these networks by reducing operator workload and applying network changes in a consistent and scalable way. The ability to do segmentation on demand and its ability to be integrated into the OT line-of-business processes and applications using REST APIs is a game-changing innovation that OT networks can utilize. Another key takeaway is that Cisco SD-Access helps implement and integrate OT networks with IT Networks and how OT Networks can leverage the new functionality and adapt to newer technologies evolved on the IT networks.