10-03-2019 09:46 AM - edited 10-04-2019 07:01 AM
This document describes how to install and configure the Cisco AnyConnect Network Visibility Module (NVM) on an end-user system using AnyConnect 4.7.x or higher as well as how to install and configure the associated Splunk Enterprise components and NVM Collector.
For more information about the solution please refer to www.cisco.com/go/cesa.
The components that make up the solution are:
The Cisco AnyConnect Network Visibility Module provides a continuous feed of high value endpoint telemetry. NVM empowers organizations to see endpoint & user behavior on their network, collects flows from endpoints both on and off-premise along with valuable context like users, applications, devices, locations and destinations. Splunk Enterprise consumes the telemetry data and provides the analytics capabilities and reports.
This technote is a configuration example for AnyConnect NVM with Splunk Enterprise as part of the new CESA solution.
Below is a high level overview of a deployment in its simplest form. This would be an all-in-one configuration running on 64-bit Linux.
This configuration is how most demonstrations will be setup and is also useful in a small production deployment.
Below is a more comprehensive set of options that are available for deployment. Typically a production setup will be distributed and have several Splunk Enterprise nodes.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Cisco AnyConnect is a unified agent that delivers multiple security services to protect the enterprise. AnyConnect is most commonly used as an enterprise VPN client, but it also supports additional modules that cater to different aspects of enterprise security. The additional modules enable security features like posture assessment, web security, malware protection, network visibility and more.
This technote is about Network Visibility Module (NVM), which integrates with Cisco AnyConnect to provide administrators the ability to monitor endpoint application usage.
For more information regarding Cisco Anyconnect, refer to:
Cisco AnyConnect Secure Mobility Client Administrator Guide, Release 4.7
IPFIX is an IETF protocol to define a standard for exporting IP flow information for various purposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, though not directly compatible. Cisco nvzFlow is a protocol specification based on the IPFIX protocol. By design, IPFIX is an extensible protocol allowing one to define new parameters to convey information. Cisco nvzFlow protocol extends the IPFIX standard and defines new Information Elements as well as defines a standard set of IPFIX templates that are conveyed as part of the telemetry used by AnyConnect NVM.
For more information on IPFIX, refer to rfc5101, rfc7011, rfc7012, rfc7013, rfc7014, rfc7015.
A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk.
Cisco provides a collector specifically designed for the nvzFlow protocol and bundled with the Splunk App.
Splunk Enterprise is a powerful tool that collects and analyses diagnostic data to give meaningful information about the IT infrastructure. It provides a one-stop location for administrators to collect data that is crucial in understanding the health of the network.
Splunk is a partner of Cisco's and the CESA solution was created in collaboration with them.
IP address conventions in this technote :
Collector IP address: 192.0.2.123
Splunk IP address: 192.0.2.113
This section covers configuration of Cisco NVM components.
AnyConnect NVM configuration is saved in an XML file that contains information about the collector IP address and port number, along with other information. The collector IP address and port number need to be correctly configured on NVM client profile.
For correct operation of the NVM module, the XML file is required to be placed in this directory:
If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed along with Anyconnect NVM deployment.
XML profile example:
<?xml version="1.0" encoding="UTF-8"?> -<NVMProfile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="NVMProfile.xsd"> -<CollectorConfiguration> <CollectorIP>192.0.2.123</CollectorIP> <Port>2055</Port> </CollectorConfiguration> <Anonymize>false</Anonymize> <CollectionMode>all</CollectionMode> </NVMProfile>
NVM profile can be created using two different tools:
This method is preferable if Anyconnect NVM is being deployed via Cisco ASA.
1. Navigate to Configuration > Remove Access VPN > Network (Client) Access > Anyconnect Client Profile
2. Click Add
3. Give the profile a name. In Profile Usage, select Network Visibility Service Profile
4. Assign it to the group-policy being used by Anyconnect users. Click OK.
5. The new policy is created. Click Edit
6. Fill information regarding the Collector IP address and port number. Click OK.
7. Click Apply.
This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM is being deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to Cisco ISE, or copied directly to endpoints.
For detailed information on Anyconnect Profile Editor, refer to:
This technote assumes that Anyconnect is already configured on the ASA, and only NVM module configuration needs to be added. For detailed information on ASA Anyconnect configuration, refer to:
ASDM Book 3: Cisco ASA Series VPN ASDM Configuration Guide, 7.5
In order to enable Anyconnect NVM module on Cisco ASA, perform these steps:
1. Navigate to Configuration > Remote Access VPN > Network (Client) Access > Group Policies
2. Select relevant group-policy and click Edit
3. Within the group-policy pop-up, navigate to Advanced > Anyconnect Client.
4. Expand Optional Client Modules to Download and select Anyconnect Network Visibility.
5. Click OK and apply changes.
Adding AnyConnect Image
Select Add > Agent Resources, and upload the Anyconnect package file.
Confirm the package's hash in the pop-up.
The file-hash can be verified against Cisco.com download page or using third-party tool.
This step can be repeated to add multiple Anyconnect images. (for Mac OSX and Linux OS)
Adding AnyConnect NVM profile
Select Add > Agent Resources, and upload the NVM client profile.
Add AnyConnect configuration file
Select Add > AnyConnect Configuration
Choose the package uploaded in previous step.
Enable NVM in the AnyConnect Module Selection along with the policy required.
In the above section, we enable AnyConnect Client modules, profiles, customization/language packages, and the Opswat packages.
For detailed information regarding web-deployment configuration on Cisco ISE, refer to:
AnyConnect NVM sends flow information only when it is on a Trusted Network. It uses the TND feature of AnyConnect client to learn if the endpoint is in a trusted network or not.
Trusted Network Detection is configured in the AnyConnect Client Profile (xml) used for VPN regardless of whether the VPN compoinent is being used in the environment or not. TND is enabled by configuring the Automatic VPN Policy section in the profile. At a minimum, a single Trusted DNS Domain or Trusted DNS Server must be populated. The actions taken by AnyConnect when the client has determined that it is on a Trusted Network can be set to 'DoNothing' mode using the pull-down for the Trusted and Untrusted Network Policy.
For additional details on TND configuration, refer to:
Configure Trusted Network Detection
Deploying Anyconnect NVM solution involves these steps:
1. Configure Anyconnect NVM on Cisco ASA/ISE
2. Set up IPFIX Collector component
3. Set up Splunk with Cisco NVM App and Add-On
This step has been covered in detail in the Configure section.
Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.
The Collector Component is responsible for collecting and translating all IPFIX data from the endpoints and forwarding it to the Splunk Add-On. The NVM collector runs on 64-bit Linux. CentOS, Ubuntu and Docker configuration scripts are included. The CentOS install scripts and configuration files can also be used in Fedora and Redhat distributions as well.
In a typical distributed Splunk Enterprise deployment, the collector should be run on either a standalone 64-bit Linux system or a Splunk Forwarder node running on 64-bit Linux.
NOTE: The solution can also be run on a single 64-bit Linux system that includes the NVM collector and Splunk Enterprise components for use in a small deployment or for demonstration purposes.
In order to install the collector you will need to copy the application in the acnvmcollector.zip file, located in the $APP_DIR$/appserver/addon/ directory to the system you plan to install it on. Extract the files on the system where you plan to install the collector on and execute the install.sh script with super user privileges. It is recommended to read the $PLATFORM$_README file in the .zip bundle before executing the install.sh script. The $PLATFORM$_README file provides information on the relevant configuration settings that need to be verified and modified (if necessary) before the install.sh script is executed. At a minimum, you will need to configure the address of the Splunk instance you will be forwarding data to. Failing to properly configure the system can cause the collector to operate incorrectly.
NOTE: Ensure that network and host firewalls are properly configured to allow the UDP traffic for the source and destination addresses and ports
A single NVM collector instance can handle a minimum of 5000 flows per second on a properly sized system. The collector needs to be configured and running before the Splunk App can be used.
By default, the collector receives flows from AnyConnect NVM endpoints on UDP port 2055.
Additionally, the collector produces three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively.
The receive and data feed ports can be changed by altering the acnvm.conf file and restarting the collector instance. Make sure that any host/network firewalls between endpoints and the collector or between the collector and Splunk system(s) are open for the configured UDP ports and addresses. Also ensure that your AnyConnect NVM configuration matches your collector configuration.
Once all components are installed and running, refer to the Help files section from within the Splunk application for detailed information about the pre-configured reports, data model and information elements that are created by the solution.
You may want to restart one of your AnyConnect endpoints and validate that data is being sent to the solution.
The information needs to be configured in the configuration file (acnvm.conf):
1. IP address and listening port of Splunk instance.
2. Listening port for collector (incoming IPFIX data).
Per Flow Data Port, Endpoint Identity Data Port, Endpoint Interface Data and Collector Port are pre-configured to default settings in the configuration file. Ensure that these values are changed if non-default ports are being used.
This information is added in the configuration file (acnvm.conf):
GNU nano 2.2.6 File: acnvm.conf { "syslog_server_ip" : "192.0.2.113", "syslog_flowdata_server_port" : 20519, "syslog_sysdata_server_port" : 20520,
"syslog_intdata_server_port" : 20521, "netflow_collector_port" : 2055, "log_level" : 7 }
For more information, refer to:
https://splunkbase.splunk.com/app/2992/#/details
Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from end points in usable reports, and correlates user and endpoint behavior.
Link for Cisco NVM App for Splunk on Splunkbase:
https://splunkbase.splunk.com/app/2992/
Link for Cisco NVM Add-On for Splunk on Splunkbase:
https://splunkbase.splunk.com/app/4221/
Navigate to Splunk > Apps and install the tar.gz file downloaded from the Splunkbase or search within the Apps section.
Next you need to install the Add-On following the same process. Confirm that both are installed by viewing Splunk Apps page:
The default configuration receives three data feeds for Splunk, Per Flow Data, Endpoint Identity Data and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively. (see Step 2)
The Add-On then maps these to Splunk sourcetypes cisco:nvm:flowdata, cisco:nvm:sysdata and cisco:nvm:ifdata.
In order to change default ports, navigate to Splunk > Settings > Data Input > UDP
After successful installation, the Network Visibility Module should be listed in Installed Modules, within in the Information section of AnyConnect Secure Mobility client.
Also, verify if the nvm service is running on the end point and profile is in the required directory.
Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflow from the endpoints at all times.
root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status * acnvmcollector is running root@ubuntu-splunkcollector:~$
Ensure that Splunk and its relevant services are running. For documentation on troubleshooting Splunk, please refer to their website.
1. IPFIX packets are generated on client endpoints by AnyConnect NVM module.
2. Client endpoints forward IPFIX packets to the Collector IP address
3. Collector collects the information and forwards it to Splunk
4. Collector sends traffic to Splunk on two different streams: Per Flow Data and Endpoint Identity Data
All traffic is UDP based on there is no acknowledgement of traffic.
Default port for traffic:
IPFIX data 2055
Per Flow Data 20519
Endpoint Data 20520
Interface Data 20521
NVM module caches IPFIX data and sends it to collector when it is in Trusted Network. This can either be when the laptop is connected to the corporate network (on-prem) or when it is connected via VPN.
IPFIX traffic as seen in Wireshark:
NVM relies on TND for detecting when the endpoint is within trusted network. If the TND configuration is incorrect, this will cause issues with NVM.
TND works based on information received via DHCP: domain-name and DNS server. If the DNS server and/or domain-name match the configured values, then the network is deemed to be trusted.
If NVM is not forwarding traffic to collector, then it could be an issue with TND.
IPFIX flow templates are sent to collector at the start of the IPFIX communication. These templates help the collector to make sense of the IPFIX data.
The collector also preloads templates to ensure that even if the client has not sent them that the data can be parsed. If a newer version of the client is released with protocol changes, the new templates sent by the client will be used.
A template is sent out under the following conditions:
In rare circumstances, a template may not be found. This can be easily remidied by restarting one of the endpoints.
The issue can be identified by observing no template found in a packet capture on the end point, or no templates for flowset in the collector logs.
Packet capture
Collector logs:
Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates for flowset 258 for 10.150.176.167 yet Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234 Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: =================> flowsetid=258 flowsetlen=218 Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates for flowset 258 for 10.150.176.167 yet
Cisco always recommends the latest software version of AnyConnect at the time of use or updating. While choosing AnyConnect version, please use the latest 4.7.x client or later. This will give the latest enhancements with respect to NVM.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: