- Introduction
- Deployment Overview
- Requirements
- Components Used
- Background Information
- Cisco AnyConnect Secure Mobility Client - More than VPN
- Internet Protocol Flow Information Export (IPFIX)
- IPFIX Collector
- Splunk Enterprise
- Topology
- Configure
- Deploy
- Install
- Verify
- Troubleshoot
- Recommended Release
- Related Links
Introduction
This document describes how to install and configure the Cisco AnyConnect Network Visibility Module (NVM) on an end-user system using AnyConnect 4.7.x or higher as well as how to install and configure the associated Splunk Enterprise components and NVM Collector.
For more information about the solution please refer to www.cisco.com/go/cesa.
The components that make up the solution are:
- Cisco AnyConnect Secure Mobility Client with Network Visibility Module (NVM) enabled
- Cisco AnyConnect Network Visibility Module (NVM) App for Splunk
- Cisco NVM Technology Add-On for Splunk
- NVM Collector (bundled in a zip file with the Splunk Application)
The Cisco AnyConnect Network Visibility Module provides a continuous feed of high value endpoint telemetry. NVM empowers organizations to see endpoint & user behavior on their network, collects flows from endpoints both on and off-premise along with valuable context like users, applications, devices, locations and destinations. Splunk Enterprise consumes the telemetry data and provides the analytics capabilities and reports.
This technote is a configuration example for AnyConnect NVM with Splunk Enterprise as part of the new CESA solution.
Deployment Overview
Below is a high level overview of a deployment in its simplest form. This would be an all-in-one configuration running on 64-bit Linux.
This configuration is how most demonstrations will be setup and is also useful in a small production deployment.
Below is a more comprehensive set of options that are available for deployment. Typically a production setup will be distributed and have several Splunk Enterprise nodes.
Requirements
Cisco recommends that you have knowledge of these topics:
- AnyConnect 4.3.x or higher with NVM
- AnyConnect APEX license
- ASDM
- Familiarity with Splunk Enterprise and how to install Splunk Apps and Add-ons
Components Used
The information in this document is based on these software and hardware versions:
- Cisco AnyConnect Security Mobility Client 4.3.x or later
- Cisco AnyConnect Profile Editor
- Cisco Adaptive Security Appliance (ASA), version 9.5.2
- Cisco Adaptive Security Device Manager (ASDM), version 7.5.1
- Splunk Enterprise 6.3 or later
- Ubuntu 14.04.3 LTS as a collector device
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
Cisco AnyConnect Secure Mobility Client - More than VPN
Cisco AnyConnect is a unified agent that delivers multiple security services to protect the enterprise. AnyConnect is most commonly used as an enterprise VPN client, but it also supports additional modules that cater to different aspects of enterprise security. The additional modules enable security features like posture assessment, web security, malware protection, network visibility and more.
This technote is about Network Visibility Module (NVM), which integrates with Cisco AnyConnect to provide administrators the ability to monitor endpoint application usage.
For more information regarding Cisco Anyconnect, refer to:
Cisco AnyConnect Secure Mobility Client Administrator Guide, Release 4.7
Internet Protocol Flow Information Export (IPFIX)
IPFIX is an IETF protocol to define a standard for exporting IP flow information for various purposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, though not directly compatible. Cisco nvzFlow is a protocol specification based on the IPFIX protocol. By design, IPFIX is an extensible protocol allowing one to define new parameters to convey information. Cisco nvzFlow protocol extends the IPFIX standard and defines new Information Elements as well as defines a standard set of IPFIX templates that are conveyed as part of the telemetry used by AnyConnect NVM.
For more information on IPFIX, refer to rfc5101, rfc7011, rfc7012, rfc7013, rfc7014, rfc7015.
IPFIX Collector
A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk.
Cisco provides a collector specifically designed for the nvzFlow protocol and bundled with the Splunk App.
Splunk Enterprise
Splunk Enterprise is a powerful tool that collects and analyses diagnostic data to give meaningful information about the IT infrastructure. It provides a one-stop location for administrators to collect data that is crucial in understanding the health of the network.
Splunk is a partner of Cisco's and the CESA solution was created in collaboration with them.
Topology
IP address conventions in this technote :
Collector IP address: 192.0.2.123
Splunk IP address: 192.0.2.113
Configure
This section covers configuration of Cisco NVM components.
AnyConnect NVM client profile
AnyConnect NVM configuration is saved in an XML file that contains information about the collector IP address and port number, along with other information. The collector IP address and port number need to be correctly configured on NVM client profile.
For correct operation of the NVM module, the XML file is required to be placed in this directory:
- For Windows 7 and later: %ALLUSERSPROFILE%\Cisco\Cisco AnyConnect Secure Mobility Client\NVM
- For Mac OSX: /opt/cisco/anyconnect/nvm
If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed along with Anyconnect NVM deployment.
XML profile example:
<?xml version="1.0" encoding="UTF-8"?> -<NVMProfile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="NVMProfile.xsd"> -<CollectorConfiguration> <CollectorIP>192.0.2.123</CollectorIP> <Port>2055</Port> </CollectorConfiguration> <Anonymize>false</Anonymize> <CollectionMode>all</CollectionMode> </NVMProfile>
NVM profile can be created using two different tools:
- Cisco ASDM
- AnyConnect Profile Editor
Configure NVM client profile via ASDM
This method is preferable if Anyconnect NVM is being deployed via Cisco ASA.
1. Navigate to Configuration > Remove Access VPN > Network (Client) Access > Anyconnect Client Profile
2. Click Add
3. Give the profile a name. In Profile Usage, select Network Visibility Service Profile
4. Assign it to the group-policy being used by Anyconnect users. Click OK.
5. The new policy is created. Click Edit
6. Fill information regarding the Collector IP address and port number. Click OK.
7. Click Apply.
Configure NVM client profile via AnyConnect Profile Editor
This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM is being deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to Cisco ISE, or copied directly to endpoints.
For detailed information on Anyconnect Profile Editor, refer to:
Configure Web-Deployment on Cisco ASA
This technote assumes that Anyconnect is already configured on the ASA, and only NVM module configuration needs to be added. For detailed information on ASA Anyconnect configuration, refer to:
ASDM Book 3: Cisco ASA Series VPN ASDM Configuration Guide, 7.5
In order to enable Anyconnect NVM module on Cisco ASA, perform these steps:
1. Navigate to Configuration > Remote Access VPN > Network (Client) Access > Group Policies
2. Select relevant group-policy and click Edit
3. Within the group-policy pop-up, navigate to Advanced > Anyconnect Client.
4. Expand Optional Client Modules to Download and select Anyconnect Network Visibility.
5. Click OK and apply changes.
Configure Web-Deployment on Cisco ISE
- In order to configure Cisco ISE for Anyconnect Web-Deployment, perform these steps:
- In Cisco ISE GUI, navigate to Policy > Policy Elements > Results
- Expand Client Provisioning to show Resources, and select Resources
Adding AnyConnect Image
Select Add > Agent Resources, and upload the Anyconnect package file.
Confirm the package's hash in the pop-up.
The file-hash can be verified against Cisco.com download page or using third-party tool.
This step can be repeated to add multiple Anyconnect images. (for Mac OSX and Linux OS)
Adding AnyConnect NVM profile
Select Add > Agent Resources, and upload the NVM client profile.
Add AnyConnect configuration file
Select Add > AnyConnect Configuration
Choose the package uploaded in previous step.
Enable NVM in the AnyConnect Module Selection along with the policy required.
In the above section, we enable AnyConnect Client modules, profiles, customization/language packages, and the Opswat packages.
For detailed information regarding web-deployment configuration on Cisco ISE, refer to:
Trusted Network Detection
AnyConnect NVM sends flow information only when it is on a Trusted Network. It uses the TND feature of AnyConnect client to learn if the endpoint is in a trusted network or not.
Trusted Network Detection is configured in the AnyConnect Client Profile (xml) used for VPN regardless of whether the VPN compoinent is being used in the environment or not. TND is enabled by configuring the Automatic VPN Policy section in the profile. At a minimum, a single Trusted DNS Domain or Trusted DNS Server must be populated. The actions taken by AnyConnect when the client has determined that it is on a Trusted Network can be set to 'DoNothing' mode using the pull-down for the Trusted and Untrusted Network Policy.
For additional details on TND configuration, refer to:
Configure Trusted Network Detection
Deploy
Deploying Anyconnect NVM solution involves these steps:
1. Configure Anyconnect NVM on Cisco ASA/ISE
2. Set up IPFIX Collector component
3. Set up Splunk with Cisco NVM App and Add-On
Step 1. Configure AnyConnect NVM on Cisco ASA/ISE
This step has been covered in detail in the Configure section.
Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.
Step 2. Set up IPFIX Collector component
The Collector Component is responsible for collecting and translating all IPFIX data from the endpoints and forwarding it to the Splunk Add-On. The NVM collector runs on 64-bit Linux. CentOS, Ubuntu and Docker configuration scripts are included. The CentOS install scripts and configuration files can also be used in Fedora and Redhat distributions as well.
In a typical distributed Splunk Enterprise deployment, the collector should be run on either a standalone 64-bit Linux system or a Splunk Forwarder node running on 64-bit Linux.
NOTE: The solution can also be run on a single 64-bit Linux system that includes the NVM collector and Splunk Enterprise components for use in a small deployment or for demonstration purposes.
In order to install the collector you will need to copy the application in the acnvmcollector.zip file, located in the $APP_DIR$/appserver/addon/ directory to the system you plan to install it on. Extract the files on the system where you plan to install the collector on and execute the install.sh script with super user privileges. It is recommended to read the $PLATFORM$_README file in the .zip bundle before executing the install.sh script. The $PLATFORM$_README file provides information on the relevant configuration settings that need to be verified and modified (if necessary) before the install.sh script is executed. At a minimum, you will need to configure the address of the Splunk instance you will be forwarding data to. Failing to properly configure the system can cause the collector to operate incorrectly.
NOTE: Ensure that network and host firewalls are properly configured to allow the UDP traffic for the source and destination addresses and ports
A single NVM collector instance can handle a minimum of 5000 flows per second on a properly sized system. The collector needs to be configured and running before the Splunk App can be used.
By default, the collector receives flows from AnyConnect NVM endpoints on UDP port 2055.
Additionally, the collector produces three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively.
The receive and data feed ports can be changed by altering the acnvm.conf file and restarting the collector instance. Make sure that any host/network firewalls between endpoints and the collector or between the collector and Splunk system(s) are open for the configured UDP ports and addresses. Also ensure that your AnyConnect NVM configuration matches your collector configuration.
Once all components are installed and running, refer to the Help files section from within the Splunk application for detailed information about the pre-configured reports, data model and information elements that are created by the solution.
You may want to restart one of your AnyConnect endpoints and validate that data is being sent to the solution.
The information needs to be configured in the configuration file (acnvm.conf):
1. IP address and listening port of Splunk instance.
2. Listening port for collector (incoming IPFIX data).
Per Flow Data Port, Endpoint Identity Data Port, Endpoint Interface Data and Collector Port are pre-configured to default settings in the configuration file. Ensure that these values are changed if non-default ports are being used.
This information is added in the configuration file (acnvm.conf):
GNU nano 2.2.6 File: acnvm.conf
{
"syslog_server_ip" : "192.0.2.113",
"syslog_flowdata_server_port" : 20519,
"syslog_sysdata_server_port" : 20520,
"syslog_intdata_server_port" : 20521,
"netflow_collector_port" : 2055,
"log_level" : 7
}
For more information, refer to:
https://splunkbase.splunk.com/app/2992/#/details
Step 3. Set up Splunk with Cisco NVM App and Add-On for Splunk
Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from end points in usable reports, and correlates user and endpoint behavior.
Link for Cisco NVM App for Splunk on Splunkbase:
https://splunkbase.splunk.com/app/2992/
Link for Cisco NVM Add-On for Splunk on Splunkbase:
https://splunkbase.splunk.com/app/4221/
Install
Navigate to Splunk > Apps and install the tar.gz file downloaded from the Splunkbase or search within the Apps section.
Next you need to install the Add-On following the same process. Confirm that both are installed by viewing Splunk Apps page:
The default configuration receives three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively. (see Step 2)
The Add-On then maps these to Splunk sourcetypes cisco:nvm:flowdata, cisco:nvm:sysdata and cisco:nvm:ifdata.
In order to change default ports, navigate to Splunk > Settings > Data Input > UDP
Verify
Validate AnyConnect NVM installation
After successful installation, the Network Visibility Module should be listed in Installed Modules, within in the Information section of AnyConnect Secure Mobility client.
Also, verify if the nvm service is running on the end point and profile is in the required directory.
Validate Collector status as Running
Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflow from the endpoints at all times.
root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status * acnvmcollector is running root@ubuntu-splunkcollector:~$
Validate Splunk
Ensure that Splunk and its relevant services are running. For documentation on troubleshooting Splunk, please refer to their website.
Troubleshoot
Packet Flow
1. IPFIX packets are generated on client endpoints by AnyConnect NVM module.
2. Client endpoints forward IPFIX packets to the Collector IP address
3. Collector collects the information and forwards it to Splunk
4. Collector sends traffic to Splunk on two different streams: Per Flow Data and Endpoint Identity Data
All traffic is UDP based on there is no acknowledgement of traffic.
Default port for traffic:
IPFIX data 2055
Per Flow Data 20519
Endpoint Data 20520
Interface Data 20521
NVM module caches IPFIX data and sends it to collector when it is in Trusted Network. This can either be when the laptop is connected to the corporate network (on-prem) or when it is connected via VPN.
Basic troubleshooting steps
- Ensure network connectivity between client endpoint and collector.
- Ensure network connectivity between collector and splunk.
- Ensure that NVM is correctly installed on client endpoint.
- Apply captures on endpoint to see if IPFIX traffic is being generated.
- Apply captures on collector to see if it is receiving IPFIX traffic, and if it is forwarding traffic to Splunk.
- Apply captures on Splunk to see if it is receiving traffic.
IPFIX traffic as seen in Wireshark:
Trusted Network Detection (TND)
NVM relies on TND for detecting when the endpoint is within trusted network. If the TND configuration is incorrect, this will cause issues with NVM.
TND works based on information received via DHCP: domain-name and DNS server. If the DNS server and/or domain-name match the configured values, then the network is deemed to be trusted.
If NVM is not forwarding traffic to collector, then it could be an issue with TND.
Flow Templates
IPFIX flow templates are sent to collector at the start of the IPFIX communication. These templates help the collector to make sense of the IPFIX data.
The collector also preloads templates to ensure that even if the client has not sent them that the data can be parsed. If a newer version of the client is released with protocol changes, the new templates sent by the client will be used.
A template is sent out under the following conditions:
- There is a change in the NVM client profile.
- There is a network change event.
- The nvmagent service is restarted.
- End point is rebooted/restarted.
In rare circumstances, a template may not be found. This can be easily remidied by restarting one of the endpoints.
The issue can be identified by observing no template found in a packet capture on the end point, or no templates for flowset in the collector logs.
Packet capture
Collector logs:
Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates for flowset 258 for 10.150.176.167 yet Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234 Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: =================> flowsetid=258 flowsetlen=218 Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates for flowset 258 for 10.150.176.167 yet
Recommended Release
Cisco always recommends the latest software version of AnyConnect at the time of use or updating. While choosing AnyConnect version, please use the latest 4.7.x client or later. This will give the latest enhancements with respect to NVM.
Related Links
- Cisco Endpoint Security Analytics on Splunk (Quick Start Guide): https://www.cisco.com/c/dam/en/us/products/se/2019/8/Collateral/endpoint-sec-analy-quickstart-guide....
- Cisco AnyConnect Network Visibility (NVM) App for Splunk: https://splunkbase.splunk.com/app/2992/
- Splunk Documentation on Splunk Collector Setup and installing collector scripts : https://splunkbase.splunk.com/app/2992/#/documentation
- Cisco AnyConnect Secure Mobility Client- Administration Guide
- Release notes of AnyConnect 4.x
- Find What Your Endpoint Anti-Malware is Missing with CESA Built on Splunk