The Project Story for this Article can be found at: ISE Deployment and Operation: Lessons from Large, Complex Environment.
- PART I
- PART II
- Preface
- Caixa Econômica Federal
- ISE
- Overview
- Use Cases & Outcomes
- Cisco Secure Client vs Cisco AnyConnect
- A Real-World Example (Challenges/Tips)
- Deployment
- Be Prepared for the Next Crisis !!!
- Periodically Reevaluate your Deployment !!!
- Unknowns to Knowns to Classified !!!
- Disaster Recovery
- A Scheduled Backup and a Functional Restore ... May Not Be Enough !!!
- The War Room
- "1st Rule"
- Your Business Behaviors
- Linux Devices Certificate Enrollment using NDES (SCEP)
- New Release / New Features
- Software
- APIs
- API for Distributed Management NADs
- API for Distributed Management and Automation of Whitelists
- Data Connect
- Sharing Knowledge - The Only Option
- Security Knowledge Base Articles
- Slow Replication
- Queue Link Error
- Ideas & Enhancements
- Conclusion
The Portuguese version of this Article can be found at: Navegando de Forma Segura num Ambiente Caótico - Parte II.
 |
For an offline or printed copy of this document, simply choose ⋮ Options > Printer Friendly Page. You may then Print > Print to PDF or Copy & Paste to any other document format you like. |
This is a two-part Knowledge Base Article:
- Navigating Security in a Chaotic Environment - Part I: Introduction to Chaotic Environment & AI (Artificial Intelligence)
Discover how to tackle Chaotic Environments head-on and explore the role of AI in navigating this complexity !!!
- Navigating Security in a Chaotic Environment - Part II: Real-World Example, Challenges & Tips
Gain insights from Real-World Examples of Chaotic Environments, uncover Challenges, and equip yourself with Tips to "conquer this War".
PART I
Link:
Navigating Security in a Chaotic Environment - Part I
PART II
Thanks @Durval Vieira , @Ivan Tratz , @Renato Guardia , @Robson Ribeiro , @Adonay dos Anjos , @Daniel Volpato for the contribution !!!
Preface
A real-world scenario with many adversities generates many Lessons Learned and Success Stories.
The Challenges & Solutions faced by a huge Customer provide all Readers with a unique opportunity to learn from the daily experience of this constant "battlefield".
Caixa Econômica Federal is a huge Financial Institution, with a very large heterogeneous Network Environment that was looking to provide Zero Trust Access to their Employees, Contractors and Devices that need access to Company Applications and Internet Access.
The main Challenge was to find Technical and Operational Solutions that would provide Visibility and Control of Users and Devices, in an existing Diverse and in some cases Legacy Infrastructure with limited Security Capabilities.
Caixa Econômica Federal
Caixa Econômica Federal is a Public Bank founded in 1861 with 85K+ Employees, 115K+ Users, and 150M+ Customers serving the Brazilian territory of 8.5M km2 and 26 States via 53K+ Service Points (among these 13 Truck Agencies and 2 Boat Agencies), 15K+ Switches and 250K+ Endpoints.
As Caixa Econômica Federal is:
- in the Public Sector ... bidding is the main reason that creates a Chaotic Environment
- is a Bank ... it's an attractive target for Cybercriminals
ISE
Overview
Cisco Secure Zero Trust is a comprehensive approach to securing all access across People, Applications and Environments. The three primary areas of Zero Trust are: WorkForce, WorkPlace and WorkLoads ...
WorkPlace is the heart of Cisco ISE !!!
Next up ... an overview of Cisco ISE in 4 images:
Use Cases & Outcomes
Use Cases & Outcomes that can enabled through Cisco ISE:
Cisco Secure Client vs Cisco AnyConnect
Cisco Secure Client is the rebranded version of Cisco AnyConnect.
Cisco Secure Client is a comprehensive Security Client that offers a suite of security services through its modular approach:
A Real-World Example (Challenges/Tips)
Our Real-World Example is based on Caixa Econômica Federal and Cisco ISE !!!
Deployment
Be Prepared for the Next Crisis !!!
Security should never be an afterthought in the Development Process, otherwise it leads to vulnerabilities and increases risks.
In a time of great uncertainty (where the only certainty is: "When will the next crisis be ?"), we must be prepared for a quick and smooth shift from Wired to VPN, the 2019 Pandemic was a great example !!!
Challenges:
- During crisis like this, rapidly and safely moving 150K+ Users to fully remote work (from Wired to VPN) while maintaining not only the Quality of Service, but also the Everyone Experience (Customer and Employee Experience) is more than a challenge ... it's a MUST ... it's the Brand Reputation !!!
Tips:
- Release & Patch
Avoid different Release and Patch between VPN and Wired ISE Cluster.
- PAN & MnT
Dedicated PAN & MnT in both Deployment to support the extra load when the shift from Wired to VPN is required.
- PSNs
Be prepared to create a new PSNs arrangement (De-Register PSNs from the Wired ISE Cluster and Register them to the VPN ISE Cluster)
- Load Balancer
Load Balancers between NADs (Switches / Firewalls) and PSNs (Wired / VPN Cluster) provide flexibility for a fast migration to the new PSNs arrangement !!!
Links:
Performance and Scalability Guide for Cisco ISE
Periodically Reevaluate your Deployment !!!
The Sizing Guidelines for ISE Deployment (Table 3 of the Performance and Scalability Guide for Cisco ISE) are derived based on tests under the following conditions:
- ISE Deployment are formed in a Single Datacenter deployed in same Region
- Low Latency (less than 5 ms) between the ISE InterNode Communications
- DOT1X Authentication and Accounting Events generated by Endpoints are in the range of 2 to 4 repetitions per day
IMPORTANT: in case of Deployments where Endpoints generate repeated Authentication and Accounting Events, more number of PSNs are required in PSN Group to help in handling Heavy Traffic.
Challenges:
- Unfortunately, in a Chaotic Environment, it is extremely difficult to predict in advance the number of repeated Authentications and Accounting Events generated by Endpoints !!!
Tips:
- periodically re-evaluating your Deployment is the ONLY option !!!
- the Suppress Repeated Failed Clients and Supress Successful Reports (at Administration > System > Settings > Protocols > RADIUS > Suppression & Reports tab) are important settings to protect the health of your Deployment !!!
- the Chart: Passed Authentication By Day (at Operations > Reports > Reports > Endpoints and Users > Authentication Summary) are an excellent chart to visualize the number of Authentications Events !!!
- always keep an eye on:
Authentication Rates (Table 5: RADIUS, Table 6: TACACS+ and Table 7: Scenario-Specific) of the Performance and Scalability Guide for Cisco Identity Services Engine:
Avg TPS (Average Transactions per Second) (at Operations > Reports > Reports > Diagnostics > Key Performance Metrics) :
TPS Line Graph (at Operations > System 360 > Log Analytics > RADIUS Authentication Summary) :
Links:
ISE - What we need to know about SNS / VM
Unknowns to Knowns to Classified !!!
The Cisco Learning Nerwork has amazing Identity Services Engine Training Videos, one of the Lessons talked about Scaling ISE Deployments for Long Term Success.
This Lesson explains that different Mediums have different Resource Consumptions as opposed to the Units of the Policy Service Node Scale (Table 4 of the Performance and Scalability Guide for Cisco ISE).
It's extremely important to go from Unknowns, to Knowns, to Classified NOT ONLY to better scale your Deployment, BUT ALSO to better protect your Deployment from Threats !!!
Challenges:
- Unfortunately, in a Chaotic Environment, with multiple "Things", it is extremely difficult to identify ALL Endpoints !!!
Tips:
- at Operations > Reports > Reports > Endpoints and Users > Authentication Summary, check the Authentication by Identity Group and search for Unknown.
- use the following tools to go from Unknowns, to Knowns, to Classified:
ISE Profiling: a feature that provides dynamic detection and classification of Endpoints connected to the Network.
Device Sensor: a feature used to gather raw Endpoint Data from Network Devices.
Links:
Configure Device Sensor for ISE Profiling
Disaster Recovery
A Scheduled Backup and a Functional Restore ... May Not Be Enough !!!
A Scheduled Backup and a Functional Restore are essential components of a Disaster Recovery strategy (or to recover from a Ransomware Attack), but they may not be enough when you have to constantly deal with Subpoena Requests and Deadlines.
Challenges:
- It's a challenge to constantly avoid wasting dozens of hours (or even days) restoring the System and obtaining the information requested in the Subpoena within the stipulated Deadline.
Tips:
- at Operations > Reports > Reports > Endpoints and Users > RADIUS Authentication | RADIUS Accounting, schedule a daily Report with the following Filter: Logged At EQUALS yesterday:
Link:
ISE - CSCwn63678 - Radius Accounting reports ISE 3.3
The War Room
To manage Business Risks during a Cybersecurity Attack, you must first identify ALL Threats and then prioritize which one have the greatest impact on your Business ...
... to identitfy ALL Threats in a Chaotic Environment with multiple Vendors, Hardware and "Things", you MUST have multiple specialized Teams ...
"1st Rule"
The "1st rule" and best practice for any War Room is to involve as few people as possible !!!
Challenges:
- Quickly calling the right specialized Teams to collaborate in a War Room and effectively respond to the Threat that have the greatest impact on your Business ... is a HUGE challenge !!!
Tips:
- at Administration > Network Resources > Network Devices populate the Location and Device Type with meaningful names (for ex.: [State] and [State]-[NAD Vendor]-[NAD Model])
- at Policy > Results > Authentication > Allowed Protocols populate the Service Name with meaningful names (for ex.: Wired-DOT1X-[NAD Vendor] or Wired-MAB-[NAD Vendor])
- at Policy > Profiling > Profiling Policies create Rules to identify your Endpoints with the Create an Identity Group for the Policy option enabled:
- at Operations > Reports > Reports > Endpoint and Users > Authentication Summary, use the following information to quickly identify the specialized Team and prioritize Business risks.
Authentication by Location
Authentication by Device Type
Authentication by Allowed Protocol
Authentication by Identity Store
Authentication by Failure Reason
Authentication by Identity Group
Your Business Behaviors
Understanding your Business Behaviors (specifics thresholds and patterns) is the 1st step to identify Anomalous Behaviors or Threats that you "cannot see" !!!
These Anomalous Behaviors can be Cybersecurity Attacks or Critical Issues that you MUST Reactive quickly, especially in a Huge Chaotic Environment !!!
Challenges:
- Quickly identify Threats that you "cannot see" is another HUGE challenge in a Chaotic Environment !!!
Tips:
- understand your Business Behaviors (specifics thresholds and patterns) using your Historical Reference Data.
- at Operations > System 360 > Log Analytics:
RADIUS Performance:
check the RADIUS All Traffic per Server:
Hour - RADIUS Authentication highlight
Hour - Pass highlight
Profiler Performance:
check the Profiler Events:
Hour - Profiler CoA highlight
Linux Devices Certificate Enrollment using NDES (SCEP)
Without a Trusted System for Authentication, malicious actors can impersonate legitimate Devices.
Digital Certificates play a crucial role in securing sensitive data and preventing unauthorized access by verifying Identities and encrypting Information.
The Enrollment process of Digital Certificates ensures that a Device obtain Digital Certificates from a Trusted Authority, known as a CA (Certificate Authority).
Microsoft NDES (Network Device Enrollment Service) is one of the role services of AD CS (Active Directory Certificate Services) that acts as a Registration Authority to enable Devices to get Certificates based on SCEP (Simple Certificate Enrollment Protocol).
Challenges:
- GPO (Group Policy) is used to distribute Digital Certificates that chain to a Trusted Root in an AD Domain to Windows Device, what about Linux Device ?
Tips:
- use NDES (SCEP)
Link:
New Release / New Features
Software
Challenges:
-
Staying up to date (Software & Hardware) is highly recommended on any Environment, but in a Huge and Chaotic Environment with multiples Devices from multiples Vendors, it's a MUST to constantly check for New Features in New Releases.
Tips:
- having an understanding of the Releases / Patches history is a MUST:
- to be Proactive, it is highly recommended to keep an eye on upcoming Releases / Patches:
- be aware of the Software Suggested Release:
Link:
Cisco Secure Client 5.1 Release Notes
Cisco Secure Client 5 Download
APIs
From Cisco ISE Release 3.1+, the MnT APIs (port 9443) the ERS APIs (port 443 or 9060) and the Open APIs (port 9070) are routed through the API Gateway (an API management solution that acts as a single entry point to multiple Cisco ISE Service APIs).
In a Distributed Environment, the Read Requests are forwarded to either a PSN or a Primary PAN, but the Write Requests are forwarded ONLY to the Primary PAN. The Primary PAN is the ONLY Node that has the write authority in the Deployment Environment.
Challenges:
- In a Huge and Chaotic Environment it is a challenge to manage the System manually.
Tips:
- create custom APIs
Link:
API for Distributed Management NADs
CODE SNIPPET for API for Distributed Management NADs
API for Distributed Management and Automation of Whitelists
CODE SNIPPET for API for Distributed Management and Automation of Whitelists
Data Connect
Data Connect is a feature that provides Read-Only access to the ISE Database so that you can query Data and create your own Reports - DIY Reports (Do It Yourself).
TCP / 2484 is used to establish Database Connections with ISE through Oracle TCPS (TCP Secure) Protocol.
| ISE 3.2+ supports Data Connect. |
Challenges:
- Defaults Reports are not always the best option to meet your needs.
Tips:
- use Data Connect to create your own custom Reports - DIY Reports (Do It Yourself).
Link:
ISE - What we need to know about Data Connect
Sharing Knowledge - The Only Option
Security Knowledge Base Articles and Ideas & Enhancements are part of the process of Sharing Knowledge and improving Cisco ISE, some examples are:
Security Knowledge Base Articles
Slow Replication
In a Chaotic Environment, Slow Replication is one of the causes of issues in an ISE Deployment.
For an in-depth analysis of Cisco ISE - Slow Replication caused by "External Actors" (not ISE), what it means , how to deal with it, and its impact on your Cisco ISE Deployment, take a look at:
Queue Link Error
ISE Messaging Service is started on each ISE Node and used for exchanging information between Nodes (via TLS using a Certificate issued by ISE's Internal CA). Queue Link is the connection between these Nodes, and Queue Link Error means that something went wrong !!!
For an in-depth analysis of Cisco ISE - Queue Link, how to deal with it, and its impact on your Cisco ISE Deployment, take a look at:
Ideas & Enhancements
The difficulties faced both on a daily basis and during a War Room become Lessons Learned and these, in turn, are translated into Ideas & Enhancements, most posted (more than 40x) in the:
Cisco Insider User Group Ideas - Security.
Like the:
others in:
Conclusion
This Knowledge Base Article took Readers on a journey from Theory (in Part I) to Experience (in Part II) to achieve the following important goals:
- explore the "theoretical universe" of a Chaotic Environment (how they arise, and how to prepare for them)
- share real-world experiences (just a few) of a Huge Chaotic Environment (challenges and tips)
- demonstrate the importance of Sharing Knowledge (the ONLY option to "win this War")
Now it's time to "get your hands dirty" and jump into Practice in your own Environment !!!
Remember that: Theory alone is not enough, Practice is just the beginning and a good Experience is made with Theory and lot of Practice !!!
We look forward to hearing from you ... feel free to share your thoughts and Lessons Learned here !!!
THANK YOU FOR YOUR TIME !!! ; )
