it / ot network design - dansk industri

Mikkel BrodersenSystems Engineer, Cisco Systems, DanmarkOktober 2018

Best Practices

IT / OT Network Design

Industrial Network Convergence

Corporate Network

Sensors and otherInput/Output Devices

Controller

Motors, DrivesActuators

SupervisoryControl

Robotics

Back-Office Mainframes andServers (ERP, MES,etc.)

OfficeApplications,Internetworking,Data Servers,Storage

Control NetworkGateway

Human MachineInterface (HMI)

Corporate Network

Sensors and otherInput/Output Devices

Motors, DrivesActuators

SupervisoryControl

Robotics

Back-Office Mainframes andServers (ERP, MES, etc.)

OfficeApplications,Internetworking,Data Servers,Storage

Human MachineInterface (HMI)

Controller

Traditional Converged Ethernet

Benefits of Industrial Ethernet in Factory Networks

Increased Visibility▪ Connectivity to devices and controllers

▪ Manufacturing—enterprise integration

Uptime and Performance▪ Security and reliability

▪ Network resiliency

Increased Efficiency▪ Standard architecture—integration and support

▪ Scalable network platform—multiple applications

Improved Event Response▪ Remote access

▪ Improved diagnostics and support

WHAT IS IT

CHALLENGE

CRITICAL NEEDS

Unified Converged Factory Network

ARCHITECTURE BUSINESS OUTCOMES

“Network issues took us hours and sometimes

days to troubleshoot. The downtime associated

with these issues was extremely costly.” - Dave

Gutshall – Harley Davidson

Complex network silos creating downtime, data isolation and vulnerabilities. Inflexible and high TCO.

• Converged Network for Flexible

Automation

• Security Built-in

• Simple

• Rapid Fault Isolation

• Resiliency

• Quality of Service

• Ease of use (NAT)

• App / Data Integration

• Ruggedized

Reduced

Downtime

Connected Factory Solution

OEE

Improvement

Connected FactoryDesigned for Digital Manufacturing

Industrial Data Center

(IDC)

Industrial

Distribution

Frame (IDF)

IDF

Physical or Virtualized Servers• FactoryTalk® Application Servers and

Services Platform• Network & Security Services – DNS,

AD, DHCP, Identity Services (AAA)• Storage Array

FactoryTalk®

Physical or Virtualized Servers• Patch Management• AV Server• Application Mirror• Remote Desktop Gateway Server

DistributionSwitch Stack

Cell/Area Zone - Levels 0–2Redundant Star Topology - Flex Links Resiliency

Unified Wireless LAN(Lines, Machines, Skids, Equipment)

Cell/Area Zone - Levels 0–2Linear/Bus/Star Topology

Autonomous Wireless LAN(Lines, Machines, Skids, Equipment)

IndustrialDemilitarized Zone

(IDMZ)

Enterprise ZoneLevels 4-5

Industrial ZoneLevels 0–3

(Plant-wide Network)

CoreSwitches

Phone

Controller

Camera

Safety Controller

Cell/Area Zone - Levels 0–2Ring Topology - Device Level Ring (DLR) Protocol


Plant Firewalls• Active/Standby• Inter-zone traffic segmentation• ACLs, IPS and IDS• VPN Services• Portal and Remote Desktop Services proxy

Level 3 - Site Operations(Control Room)

HMI

Active

AP

SSID

5 GHzWGB

SafetyI/O

Controller

WGB

LWAP

SSID

5 GHzWGB

LWAP

Controller

LWAP

SSID

2.4 GHz

Standby

Wireless

LAN Controller

(WLC)

Cell/Area Zone

Levels 0–2

Cell/Area Zone

Levels 0–2

Drive


Wide Area Network (WAN)

Data Center - Virtualized Servers• ERP - Business Systems

• Email, Web Services

• Security Services - Active Directory (AD), Identity Services (AAA)

• Network Services – DNS, DHCP

• Call Manager

Enterprise

Identity Services

Identity Services

External DMZ/ Firewall

AccessSwitches

AccessSwitches

IFW

IFW

Drive I/O Drive I/O

I/O I/O I/O RobotServoDrive

Internet

Cloud

Cloud

Cloud

Thin Client Thin Client

Connected Plantwide Ethernet Architectures

Built on Industry Standards

Remote Gateway Services

Patch Management AV Server

Application Mirror Web Services Operations ApplicationServer

EnterpriseSecurity Zone

IndustrialDMZ

Firewall

ISA95/Purdue Reference Model

7

Batch

Control

Cell/Area Zone

Discrete

Control

Drive

Control

Continuous

Process

Control

Safety

Control

Sensors Drives Actuators Robots

FactoryTalkClient

Operator Interface

FactoryTalkClient

Engineering Workstation

Operator Interface

FactoryTalkApplication

Server

FactoryTalkDirectory

Engineering Workstation

Remote Access Server

IndustrialZone

Firewall

Site Operationsand Control

Area Supervisory

Control

Basic Control

Process

Enterprise Network

Site Business Planning and Logistics NetworkE-Mail, Intranet, etc.

Level 0

Level 1

Level 2

Level 3

Level 4

Level 5

http://images.google.com/imgres?imgurl=http://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/US_Department_of_Homeland_Security_Seal.svg/360px-US_Department_of_Homeland_Security_Seal.svg.png&imgrefurl=http://commons.wikimedia.org/wiki/Image:US_Department_of_Homeland_Security_Seal.svg&h=359&w=360&sz=82&hl=en&start=1&usg=__g68ZxZ1Czwnbm-C8k0D2deynJXk=&tbnid=fqr03iu-bFLMmM:&tbnh=121&tbnw=121&prev=/images?q=department+of+homeland+security&gbv=2&hl=en

Logical ArchitectureBuilt on Industry Standards

Enterprise Zone

DMZ

Manufacturing Zone

Cell/Area Zone

Enterprise Network

Site Business Planning and

Logistics Network

Site Manufacturing Operations

and Control

Area Control

Basic Control

Process

Demilitarized Zone—

Shared Access

Level 5

Level 4

Level 3

Level 2

Level 1

Level 0

Converged Plantwide Ethernet (CPwE)

Physical or Virtualized Servers• FactoryTalk Application Servers

and Services Platform

• Network & Security Services –DNS, AD, DHCP, Identity Services (AAA)

• Storage Array

Remote AccessServer



HMI






(IDMZ)


Industrial Ethernet Switch



Controller

Safety Controller

Robot

Soft Starter

Cell/Area Zone - Levels 0–2Ring Topology - Resilient Ethernet Protocol (REP)


I/O


SafetyI/O

ServoDrive

Instrumentation


HMI

Active

AP

SSID

5 GHz

WGB

SafetyI/O

Controller

LWAP

SSID

5 GHzWGB

LWAP

Controller

LWAP

SSID

2.4 GHz

Standby

Wireless

LAN Controller

(WLC)

Cell/Area Zone

Levels 0–2

Cell/Area Zone

Levels 0–2

Drive





• Security Services - Active Directory (AD),

Identity Services (AAA)


• Call Manager

Enterprise

Identity Services

Identity Services


Internet

AccessSwitches

AccessSwitches

Reference Architecture

ASA 5500

Core

ASA 5500

IFWIFW

IFW

Converged Plantwide Ethernet (CPwE)

Physical or Virtualized Servers• FactoryTalk Application Servers

and Services Platform

• Network & Security Services –DNS, AD, DHCP, Identity Services (AAA)

• Storage Array

Remote AccessServer



HMI






(IDMZ)


Industrial Ethernet Switch



Phone

Controller

Camera

Safety Controller

Robot

Soft Starter

Cell/Area Zone - Levels 0–2Ring Topology - Resilient Ethernet Protocol (REP)


I/O


SafetyI/O

ServoDrive

Instrumentation


HMI

Active

AP

SSID

5 GHz

WGB

SafetyI/O

Controller

WGB

LWAP

SSID

5 GHzWGB

LWAP

Controller

LWAP

SSID

2.4 GHz

Standby

Wireless

LAN Controller

(WLC)

Cell/Area Zone

Levels 0–2

Cell/Area Zone

Levels 0–2

Drive





• Security Services - Active Directory (AD),

Identity Services (AAA)


• Call Manager

Enterprise

Identity Services

Identity Services


Internet

AccessSwitches

AccessSwitches

Reference Architecture

ASA 5500

Core

ASA 5500

IFWIFW

IFW

EtherNet/IPPROFINET (Industrial Protocols)

Real–Time Control

Fast Convergence

Traffic Segmentation and Management

Ease of Use

Site Operations and Control

Multi-Service Networks

Network and Security Management

Routing

Application and Data share

Access Control

Threat Protection

Enterprise/IT Integration

Collaboration

Wireless

Application Optimization

Connected Factory Reference ArchitecturesConverged Plantwide Ethernet (CPwE)

• Tested, validated and documented reference architectures

• Developed from use cases - customer and application

• Tested for performance, availability, repeatability, scalability and security

• Comprised of Cisco® and

Rockwell Automation® Validated Designs

• Built on technology and industry standards

• “Future-ready” network design

• Content relevant to both OT and IT Engineers

• Deliverables

• Recommendations, best practices, design and

implementation guidance, documented test results and configuration settings

• Simplified design, quicker deployment, reduced risk in deploying new technology

Networking Best Practices – Cell/Area ZoneBest Practices For Reducing Latency and Jitter, and to Increase Data Availability, Integrity and Security

• IP Multicast Control

• IGMP Management

• Segmentation

• Virtual LANs (VLANs)

• Prioritization

• Quality of Service (QoS)

• Apply Resiliency Protocols and multi-path topologies

• Use Fiber-media uplinks for fast convergence

• Defense-in-Depth Security

Cell/Area Zone Overview

Cell/Area Zone - Functional Area of a Production Facility. Considerations Include:

• Environmental constraints

• Range of device intelligence

• Time-sensitive applications

VFD

HMI

ControllerController

Distributed IO

ControllerController

HMI

Cell/Area ZoneCell/Area Zone

Media and

Connectors

Layer 3

Distribution

SwitchLayer 2 Access

Switch

Level 2

HMI

Level 0

Device (Drive)

Layer 2 Interswitch

Uplink-VLAN Trunk,

Layer 2 Resiliency

Layer 2 Access

Link-Single VLAN

Assigned to Port

Legend:

Level 1

ControllerIE2K / IE4K

IE5K

IE2K / IE4K IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

Typical Cell/Area Zone Traffic Flows

Engineering Laptop

Network

Management

HMIHMI

Drive Cell/Area Zone Cell/Area Zone

Manufacturing Zone

IDMZ

ControllerCIP Explicit - Informational control and administration

Intra- and inter-cell/area zone traffic flow

Non-critical administrative or data traffic using TCP

~1500 Bytes, infrequent

Above 500 ms

CIP Implicit - Producers & Consumer

>80% local

Cyclical I/O traffic, UDP unicast and multicast

<500 Bytes, Frequent

0.5 to 10’s of ms, typically 20 ms

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4KIE2K / IE4K

IE2K / IE4K

IE2K / IE4K

Resiliency for Industrial ApplicationsSupporting Multiple Topologies

• Ring Convergence

• Resilient Ethernet Protocol (REP)

• Achieves ~50 ms convergence in large, complex networks

• Redundant Star Convergence

• Multiple protocol options

• Convergence times of <100ms for Flexlinks and Etherchannel

• Tested with Rockwell applications and multicast traffic

• Fast convergence avoids application reset and improves uptime

• Critical for industrial applications

SiSi

CZ-3750

Before (FlexLink Up)

After (FlexLink Up)

Flexlink Standby

Disruption(FlexLink Down)

Schneider EcoStructure Building Network

Linear Ring Redundant Star

Cabling Requirements

Ease of Configuration

Implementation Costs

Bandwidth

Redundancy and Convergence

Disruption During Network Upgrade

Readiness for Network Convergence

Overall in Network TCO and Performance Worst OK Best

Industrial Network TopologiesCell/Area Zone Topology Options

17

Star/Bus Linear

Cell/Area Zone

Controllers, Drives, and Distributed I/O

HMI

Controllers

IE5K (Distribution Switch)

HMI

CiscoCatalyst 2955

Cell/Area ZoneControllers, Drives, and Distributed I/O

Cell/Area Zone

HMI

Controller

Redundant StarFlex LinksEtherChannel

Cell/Area Zone

Controllers, Drives, and Distributed I/O

HMI

Controllers

RingResilient EthernetProtocol (REP)



IE2K / IE4K

IE2K / IE4KIE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

IE2K / IE4K

ResiliencyProtocol

Mixed Vendor

RingRedundant

StarNet Conv>250 ms

Net Conv50-100 ms

Net Conv< 0~10 ms

Layer 3 Layer 2

STP (802.1D)

RSTP (802.1w)

MSTP (802.1s)

PVST+

REP

EtherChannel(LACP 802.3ad)

MRP (IEC 62439-2)*

Flex Links

PRP/HSR (IEC 62439)*

DLR(IEC & ODVA)

StackWise

HSRP

VRRP(IETF RFC 3768)

Network Resiliency ProtocolsSelection is Application Driven

Process and Information

Time Critical

Loss Critical

* Not part of CPwE 18

Increase ResilienceSecurely Connect Simplify Operations

Security - an enabler of IoT and ISA 99

• Increased resiliency

• Integrated OT/IT security

• Secure connectivity

• Unmanaged switch replacement

• Industrial threat protection

• Simplified compliance

Enterprise Internet

IE 5000s

Industrial Zone

Level 3

Industrial DMZ

Enterprise Zone

Levels 4 and 5

ISA 3000 or FP2100ISA 3000

IE 1000

IE 1000s

IE 4010

Cell Area Zone

Levels 0–2

IE 1000s

IE 4K’s IE 4K’s

FP2100/4100

• Device hardening with 802.1AR and ACT2

security chip

• Network hardening tools

• Certificate-based identities, user names &

passwords

• Role based Access Control

• 802.1x-based access control for meters, routers,

grid devices

• Link-layer encryption in RF Mesh

• Group-based key generation and management

(mesh)

• Network-layer encryption for WAN Backhaul

(IPSec)

Time-stamped logs, correlation at

SIEM

Separation of AMI vs. non-AMI

traffic, segmentation

Mobile Workforce

FAN Aggregation Layer within

Substation Automation

Network

Neighborhood Area Network

(RF/PLC Mesh)

RF/PLC Devices

AMI/DA Head-End

NMS HES

AAA Server

Certificate

Authority

Intrusion

Prevention

Directory

Services

SIEM

Security Services

Secure storage for encryption keys

Secure encryption keys

Network-layer encryption (IPSec)

Link-layer encryption (AES-128)

Field Area

Router (FAR)

CGR 1000

Series

Public or Private

WAN

Security

Secure Device Identity via

Digital Certificates

Strong user identities with Role-

Based Access

Security Architecture for IoT

Network Segmentation

OT User

Assigns a set of assetsto Group Cell-1 in IND

IND Topology ScreenIdentity Services

Engine

pxGrid update with asset endpoint identities and group Cell-1as custom attribute

SGT

dACL

VLAN

• Default Auth policy on ISE for switchport is configured as “open access” – i.e no NAC blocking

• PxGrid attribute “Cell-1” matches a Profiling policy on ISE and triggers corresponding Authorization policy

• ISE Authorization policy can be used to dynamically apply dACL, SGT or VLAN to switchports to segment the assets

• OT user and IT user are working with asset identities rather than IP addresses

IT UserC E L L - 1 S e g m e n t

OT User

Enabling IT-OT partnership to secure the OT network

IndustrialNetwork Director

Modbus

CIP

PROFINET

BACNet

Operational Environment

V I S I B I L I T Y

IO

PLC

DRIVE

CONTROLLER

IE Switching

NGFW

Stealthwatch

SGACL Segmentation

SGT Firewall Rules

Context based Host Groups

C O N T E X T

SXP

SGTdACL

pxGrid

C O N T E X T

C O N T E X T

ISE

pxGrid

IT / Security

REST API

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Manufacturer Usage Description

How do we secure all this things?

IoT Device Business Challenges

Device Visibility Intent-basedPolicy

Standardbased

Do you know devices well enough to

differentiate service?

Does customer knowsbehavior of devices

to build their policy?

Is there any industry standard way of connecting IoT

devices to enterprise network?

24

MUD Ecosystem Architecture

ISE receives MUD URI,

Groups device and sends MUD

URI to IOT Controller

Device emits a URL using

DHCP, LLDP, or through

802.1x

IOT

Controller

MUD-URL in

LLDP/DHCP /

802.1X

MUD-URI

in Radius/DHCP

SwitchesCritical IoT Asset

ManufacturerIOT Controller

fetches file over

HTTPS

ISE

IOT controller receives MUD File.

ISE pushes ACL/SGACL to network

devices

IOT Controller talks to MUD

File server, downloads MUD

File and applies policy on ISE

Network devices implement

and enforce desired policies

ISE pushes DACL/SGACL

(policies) to network devices

MUD File

Server

MUD-URI

Phase 1: Visibility

Phase 2: Policy

Phase 3: Trusted Introduction

DevNet site commissioned to support developers

Intro to MUD Developers Guide • Sample Code• MUD Maker Tool

Coming soon: SandBox facility and MUD file hosting and serving service https://developer.cisco.com/site/mud/

https://developer.cisco.com/site/mud/

27

BenefitsCustomer • Reduces threat surface of exploding number of devices

• Almost no additional CAPEX

• Standard approach to determining manufacturer intent

• Eases and scales access management decisions

Manufacturer• Reduces manufacturer product risk at almost no cost

• Will increase customer satisfaction and reduce support costs

• Avoids the front page

• Standards-based approach

Cisco Validated Design (CVD)

it / ot network design - dansk industri

Documents