Show Name:   A Sneak Peek Inside Talos with Geoff Serrao Contributors:  Kevin Klous, Technical Leader, Cisco Foster Lipkey, Technical Leader, Cisco Justin Roberts, Technical Leader, Cisco Geoff Serrao, Research Engineer, Cisco Talos Posting Date:    June 2019 Description:  The podcast team sits down with talented and experienced Talos Research Engineer Geoff Serrao to talk about major vulnerabilities such as Eternal Blue, Heartbleed, and Shellshock.  We also discuss persistent threats and how extensive Talos research and testing works tirelessly and around the clock to keep Cisco customers safe across the globe.  Listen Now    (MP3 40.3 MB; 37:49 mins)  Subscribe to the Podcast in iTunes by clicking the image below:   Talos: https://www.talosintelligence.com/   Vulnerabilities Mentioned: Eternal Blue https://nvd.nist.gov/vuln/detail/CVE-2017-0143 Heartbleed https://nvd.nist.gov/vuln/detail/CVE-2014-0160 ShellShock https://nvd.nist.gov/vuln/detail/CVE-2014-6271   ... View more

Show Name:   Firepower 6.4 and Other Ramblings   Contributors:    Magnus Mortensen, Technical Leader, Cisco Kevin Klous, Technical Leader, Cisco Foster Lipkey, Technical Leader, Cisco Justin Roberts Technical Leader, Cisco   Posting Date:    May 2019   Description:  The podcast team welcomes two fellow NGFW Technical Leader newcomers to the program in Foster Lipkey and Justin Roberts.  From the Cisco office in Maryand, we discuss new feature and changes in the Firepower 6.4 release as well as some upcoming news about Cisco Live 2019 in San Diego.   Listen Now    (MP3 27 MB; 31:48 mins)  Subscribe to the Podcast in iTunes by clicking the image below: Show Notes  Firepower 6.4 Release Notes: https://www.cisco.com/c/en/us/td/docs/security/firepower/640/relnotes/firepower-release-notes-640/features.html ... View more

Show Name:   A Discussion on Cisco Encrypted Traffic Analytics (ETA) with the Experts   Contributors:  Kevin Klous, Technical Leader, Cisco David White Jr., Principal Engineer, Cisco Matt Robertson, Principal Technical Marketing Engineer, Cisco Darrin Miller. Distinguished Technical Marketing Engineer, Cisco   Posting Date:    August 2018   Description:  In the Cisco Live US 2018 speaker room: The podcast team steals a few minutes from Cisco ETA and Stealthwatch experts Matt Robertson and Darrin Miller to discuss the basics of the technology and how it is helping organizations in detecting malicious content in network traffic as it increasingly goes dark (becomes encrypted).   Listen Now    (MP3 60 MB; 25:02 mins)  Subscribe to the Podcast in iTunes by clicking the image below: Quotes from the Pros: "The reality is that the networks are encrypted and threats are actually happening in those environments. We need to be able to detect threats inside of encrypted traffic. It's not really scalable to do inline decryption on everything. That's what the ETA solution was designed to do--[answer] how do we detect threats without decrypting traffic?" - Matt Robertson, Principal Technical Marketing Engineer, Cisco   "Every security architect I deal with is always saying, 'How do I turn something into an actionable event?'. That is what I really think ETA inside of Stealthwatch does...it allows us to turn all this data into actionable events." - Darrin Miller, Distinguished Technical Marketing Engineer, Cisco Show Notes  How ETA works: 3 Major Components   1. Netflow Enhancements to carry additional markers to aid in malicious traffic detection 2. Cisco Stealthwatch Enterprise - Collector, aggregator, and analyzer of network telemetry (Netflow data) 3. Cloud-hosted analytics engine. Multi-layer machine learning engine that leverages the global risk map and correlates with your organization and how it interacts with those risks.   Products Mentioned:   Encrypted Traffic Analytics (ETA) https://www.cisco.com/c/en/us/solutions/enterprise-networks/enterprise-network-security/eta.html   At-a-glance: https://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/enterprise-network-security/at-a-glance-c45-740079.pdf   Stealthwatch Enterprise https://www.cisco.com/c/en/us/products/security/stealthwatch/index.html   Cisco Stealthwatch and Cisco ISE At-a-Glance https://www.cisco.com/c/dam/en/us/products/collateral/security/stealthwatch/ise-stealthwatch-aag.pdf ... View more

Show Name: Thoughts on Security at Cisco Live US 2018 in Orlando Contributors: Kevin Klous, David White Jr., Aaron Woland, Jeff Fanelli Posting Date:  June 2018 Description: The team goes on-site in the Cisco Live Speaker room in Orlando to pick the brains of some of the best in the business.  Cisco Principal Engineers Aaron Woland and Jeff Fanelli fill us in on what's new in the Cisco Security space and give an inside look at the life of a Cisco Live Speaker.   Listen Now    (MP3 90 MB; 37:27 mins)  Subscribe to the Podcast in iTunes by clicking the image below:   Show Notes  Cisco Live US 2018 - June 10-14 in Orlando, FL, USA https://www.ciscolive.com/us/   Important Links for More Information:   Cisco Rapid Threat Containment https://www.cisco.com/c/en/us/solutions/enterprise-networks/rapid-threat-containment/index.html   Apple and Cisco Partnership https://blogs.cisco.com/security/apple-and-cisco-partnering-to-deliver-the-deepest-visibility-and-security-control-for-a-mobile-work-force   Cisco Security Connector https://www.cisco.com/c/en/us/products/security/security-connector/index.html   ... View more

  Show Name: ASA/FTD Troubleshooting Enhancements and Cisco Live US 2018 Contributors: Kevin Klous, Jay Johnston, and Magnus Mortensen Posting Date:  June 2018 Description: The team discusses the recently released troubleshooting enhancements to ASA/FTD packet tracer and packet capture tools, some new facts about FTD 6.2.3, and a look ahead to Cisco Live US 2018 coming up in Orlando, Florida. Listen Now    (MP3 82.4 MB; 34:17 mins)  Subscribe to the Podcast in iTunes by clicking the image below:   Show Notes  Cisco Live US 2018 - June 10-14 in Orlando, FL, USA https://www.ciscolive.com/us/   Sessions Mentioned:   BRKSEC-3020 - Troubleshooting ASA Firewalls https://www.ciscolive.com/us/learn/sessions/session-catalog/?search=BRKSEC-3020&showEnrolled=false TECSEC-3004 - Troubleshooting Firepower Threat Defense like a TAC Engineer (Additional cost) https://www.ciscolive.com/us/learn/sessions/session-catalog/?search=TECSEC-3004&showEnrolled=false   New Commands Discussed and Examples:   Packet Tracer with 'transmit' option  asa# sh cap capture capin type raw-data trace interface inside [Capturing - 0 bytes]   match icmp any any capture capout type raw-data trace interface outside [Capturing - 0 bytes]   match tcp any any eq ssh   asa# packet-tracer input inside tcp 10.1.1.20 10000 10.1.2.100 22 transmit  Phase: 1 Type: CAPTURE Subtype: Result: ALLOW Config: Additional Information: MAC Access list   Phase: 2 Type: ACCESS-LIST Subtype: Result: ALLOW Config: Implicit Rule Additional Information: MAC Access list   Phase: 3 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: asa# sh cap capout   1 packet captured      1: 07:48:37.428978       10.1.1.20.10000 > 10.1.2.100.22: S 2316587550:2316587550(0) win 8192 1 packet shown asa#   Packet-tracer to simulate inbound VPN traffic with 'decrypted' option:  asa# packet-tracer input inside tcp 10.1.1.20 10000 10.1.2.100 22 decrypted   ********************************************************************* WARNING: An existing decryption SA was not found. Please confirm the IPsec Phase 2 SA or Anyconnect Tunnel is established. *********************************************************************   Note:  Above output is seen when there are no active VPN Security Associations (SA) active on the ASA   Packet Capture of decrypted traffic using the 'include-decrypted' option   asa# cap capout interface outside include-decrypted match tcp any any eq 22   Important Links for More Information:   Firepower 6.2.3 Release Notes https://www.cisco.com/c/en/us/td/docs/security/firepower/623/relnotes/Firepower_Release_Notes_623.html   ASA 9.9.2 Release Notes https://www.cisco.com/c/en/us/td/docs/security/asa/asa99/release/notes/asarn99.html   ... View more

Table of Contents      Introduction      Prerequisites           Requirements           Supported platforms           Components Used           Logical Flow Diagram      Configuration           Configure LDAP           Configure AAA           Configure IP Admission           Enable IP admission                Exempting Inside Hosts from Authentication           Enable the HTTP server on the ISR           Configure CWS Redirection           Complete Sample Configuration:                LDAP                AAA                IP Admission                HTTP Server                Content-Scan / CWS           Determine the Distinguished Name Objects in Active Directory - ADSI Edit           Authentication Methods                Active NTLM                Transparent NTLM                Basic Authentication (via HTTP in clear text)                Passive NTLM                Complete Message Sequence for Active NTLM Authentication:      Troubleshooting           Show commands           Debug commands           Common Problems                IP Admission is not Intercepting HTTP requests                Users see a “404 Not Found” error in their browsers when redirected for authentication                User is Prompted for Authentication but Authentication Always Fails           Troubleshooting LDAP                The High-Level Steps of LDAP Authentication                LDAP Debug Output Analysis                RFC 4511 - Lightweight Directory Access Protocol (LDAP): The Protocol      Known Caveats                #anc40CSCum57928 - Scansafe needs a way to authenicate with HTTPS connection.                #anc41CSCul22010 - High CPU Utilization when ip http secure server is enabled with NLTM                #anc42CSCuc91949 - Scansafe IOS - Redirected HTTP site generating Tracebacks                #anc43CSCum76723 - ENH: CWS-LDAP/NTLM auth should support multiple domains and forests Introduction Many administrators who have installed ISR G2 routers that do not have ASAs in their networks choose to utilize the ISR’s Cloud Web Security (Also known as CWS and formerly ScanSafe) redirection functionality in order to take advantage of the CWS solution for web filtering.  As part of that solution, most administrators also want to utilize their existing Active Directory infrastructure to send user identity information to the CWS towers for purposes of enforcing user or group-based policies in their web filtering policies in the CWS portal. The overall concept is very much like the integration between the ASA and CDA with a few differences, the most notable being that the ISR does not actually maintain a user-to-IP mapping database and therefore users must pass through some type of authentication in order to transit the ISR and send user/group information to the CWS portal.   Please refer to the “Authentication Methods” section for more information on differences between the various authentication methods available. While the CWS redirection portion of the configuration is relatively straightforward, we have seen that many experience problems configuring the authentication piece of the solution which works using the ‘ip admission’ command that references LDAP servers and AAA authentication statements that also must be configured.  The purpose of this guide is to provide network operators with a one-stop source to reference when configuring or troubleshooting the IP admissions and LDAP portions of this solution on ISR G2 routers. Prerequisites While the IP admission and LDAP configuration could be used simply for authentication proxy on the ISR router, it is typically used in conjunction with the CWS redirection feature.  As such, this guide is intended as a reference point to supplement other CWS redirection configuration/troubleshooting documentation on ISRs. Requirements ISR must be running code version 15.2(1)T1 or later. Available in IOS (universal) images with security feature set (SEC) license. Supported platforms ISR G2 platforms:  800, 1900, 2900, and 3900 series ISRs. G1 platforms are not supported: 1800, 2800, 3800 series routers. Components Used Client workstation on Active Directory domain or capable of performing active authentication via web-browser Internet Explorer, Google Chrome, Mozilla Firefox (requires additional configuration for transparent NTLM authentication) Cisco ISR G2 router Windows Active Directory Domain Controller (AD DC) Cisco Cloud Web Security subscription Logical Flow Diagram Configuration Configure LDAP Configuring the Lightweight Directory Access Protocol (LDAP) properties of the AAA server(s): 1.    Configure an LDAP attribute map to force the username entered by the user to match the        “sAMAccountName” property in Active Directory: kklous-881(config)#ldap attribute-map ldap-username-map map type sAMAccountName username kklous-881(config-attr-map)#map type sAMAccountName username Note: We configure it in this way since the sAMAccountName attribute is unique value in AD unlike the ‘cn’ or common name attribute which is otherwise used to match by default.  For example, there can be multiple instances of “John Smith” in AD but there can only be one user with the sAMAccountName of “jsmith” which is also the user’s account logon.  Other “John Smith” accounts would have sAMAccountNames of “jsmith1, jsmith2” etc. You can use the ‘show ldap attributes’ command to view a list of LDAP attributes and their associated AAA attributes. 2.   Configure the LDAP server group: kklous-881(config)#aaa group server ldap LDAP_GROUP kklous-881(config-ldap-sg)#server DC01 3.   Configure LDAP servers kklous-881(config)#ldap server DC01 kklous-881(config-ldap-server)# ipv4 14.36.104.150 kklous-881(config-ldap-server)#attribute map ldap-username-map kklous-881(config-ldap-server)# bind authenticate root-dn CN=Csco_Service,CN=Users,DC=kklous,DC=cisco,DC=com password Cisco12345!    kklous-881(config-ldap-server)#base-dn DC=kklous,DC=cisco,DC=com            kklous-881(config-ldap-server)#search-filter user-object-type top kklous-881(config-ldap-server)#authentication bind-first This configuration generally should not be much different unless there is a need to implement a custom search-filter.  Administrators that are well-versed in LDAP will typically perform this and thus should know how to properly input this information.  If you do not know what to use as a search filter, simply use the above filter as it will locate users in a normal Active Directory environment. The part of the LDAP configuration that is often troublesome is with the distinguished names (DNs) that are required in the ‘bind-authenticate-root-dn’ and ‘base-dn commands’ which must be entered exactly as they appear in the LDAP server or the LDAP queries will fail.  In addition, the base-dn must be the lowest part of the LDAP tree where all the users being authenticated reside.  In other words, if we modified the base-dn in the above configuration to the following: base-dn OU=TestCompany,DC=kklous,DC=cisco,DC=com Then our query for users that are in “CN=Users,DC=kklous,DC=cisco,DC=com” will return no results since we would only be searching the TestCompany organizational unit (OU) and the child objects within it.  As a result, authentication would always fail for those users until they were either moved into the TestCompany OU, its subtree, or if the base-dn were changed to include that in the query.  See the "Determine the Distinguished Name Objects in Active Directory" section for details on determining the proper DNs for their base and root commands. Configure AAA Now that the LDAP servers are configured, we simply need to reference them in our AAA statements that will be used by the IP admission process: kklous-881(config)#aaa authentication login SCANSAFE_AUTH group LDAP_GROUP kklous-881(config)#aaa authorization network SCANSAFE_AUTH group LDAP_GROUP (Note:  If the above commands are not available then you may need to enter the ‘aaa new-model’ command to enable this AAA functionality as it is not enabled by default.) Configure IP Admission The IP admission piece triggers a process that prompts the user for authentication (or performs transparent authentication) and then performs LDAP queries based on the user credentials and AAA servers defined in the configuration.  If the users are authenticated successfully, the user identity information is then pulled by the content-scan process and sent out to the CWS towers along with the redirected flow.  The IP admission process is not activated until the ‘ip admission name’ command is entered on the ingress interface of the router.  For that reason, this part can be implemented without any traffic impact. kklous-881(config)#ip admission virtual-ip 1.1.1.1 virtual-host ISR_PROXY kklous-881(config)#ip admission name SCANSAFE_ADMISSION ntlm kklous-881(config)#ip admission name SCANSAFE_ADMISSION method-list authentication SCANSAFE_AUTH authorization SCANSAFE_AUTH Enable IP admission Note: This will force users to be authenticated which will cause traffic flow interruption if authentication fails . kklous-881(config)#int vlan301 (internal LAN-facing interface) kklous-881(config-if)#ip admission SCANSAFE_ADMISSION Exempting Inside Hosts from Authentication Some administrators may wish to exempt some inside hosts from being subjected to authentication for various reasons.  For example, it may be undesirable for for internal servers or devices that are not capable of NTLM or basic authentication to be affected by the IP admissions process.  In these instances, an ACL can be applied to the IP admission configuration to prevent specific host IPs or subnets from triggering IP admission. In the example below, we exempt the inside host 14.36.104.150 from being required to authenticate while requiring authentication for all other hosts: kklous-881(config)#ip access-list extended NO_ADMISSION kklous-881(config-ext-nacl)#deny ip host 14.36.104.150 any kklous-881(config-ext-nacl)#permit ip any any kklous-881(config)#ip admission name SCANSAFE_ADMISSION ntlm list NO_ADMISSION Enable the HTTP server on the ISR This is required to intercept HTTP sessions and initiate the authentication process Ip http server Ip http secure-server (only if redirection to HTTPS for authentication is required by customer) Configure CWS Redirection Below is a basic summary configuration for CWS redirection. parameter-map type content-scan global server scansafe primary name proxy139.scansafe.net port http 8080 https 8080 server scansafe secondary name proxy187.scansafe.net port http 8080 https 8080 license 0 DE749585HASDH83HGA94EA8C369 source interface Vlan302        user-group DEFAULT_GROUP username DEFAULT_USER    server scansafe on-failure allow-all interface Vlan302 (egress interface towards internet) content-scan out Complete Sample Configuration: LDAP aaa group server ldap LDAP_GROUP server DC01 ldap attribute-map ldap-username-map map type sAMAccountName username ldap server DC01 ipv4 14.36.104.150 attribute map ldap-username-map bind authenticate root-dn CN=Csco_Service,CN=Users,DC=kklous,DC=cisco,DC=com password Cisco12345! base-dn dc=klous,dc=cisco,dc=com search-filter user-object-type top authentication bind-first AAA aaa new-model aaa authentication login SCANSAFE_AUTH group LDAP_GROUP aaa authorization network SCANSAFE_AUTH group LDAP_GROUP IP Admission ip admission virtual-ip 1.1.1.1 virtual-host ISR_PROXY ip admission name SCANSAFE_ADMISSION ntlm ip admission name SCANSAFE_ADMISSION method-list authentication SCANSAFE_AUTH authorization SCANSAFE_AUTH interface Vlan301 ip admission SCANSAFE_ADMISSION HTTP Server ip http server Content-Scan / CWS parameter-map type content-scan global server scansafe primary name proxy139.scansafe.net port http 8080 https 8080 server scansafe secondary name proxy187.scansafe.net port http 8080 https 8080 license 0 DE13621123456789607EA8C369 source interface Vlan302 user-group DEFAULT_GROUP username DEFAULT_USER server scansafe on-failure allow-all interface Vlan302 content-scan out Determine the Distinguished Name Objects in Active Directory - ADSI Edit If you need to browse your Active Directory structure in order to look up distinguished names to use for your User or Group search base, you can use a tool called ADSI Edit that is built into Active Directory Domain Controllers. In order to open ADSI Edit, choose Start > Run on your Active Directory Domain Controller and enter adsiedit.msc. Once you are in ADSI Edit, right-click any object (such as an organizational unit (OU), group, or user) and choose Properties in order to view the distinguished name of that object. You can then easily copy and paste the string to your router configuration in order to avoid any typographical errors. See this screenshot for more specifics on this process: Authentication Methods There are four different types of authentication available using IP admission and these methods are often misunderstood, especially the difference between transparent and passive NTLM.  Please see the following sections for specifics and differences between the different types of authentication: Active NTLM Prompts users for authentication when transparent NTLM authentication fails.  This is usually due to the fact that the client’s browser does not support integrated Windows authentication or because the user logged into the workstation with local (non-domain) credentials.  Active NTLM authentication performs LDAP queries to the domain controller to ensure that the provided credentials are correct. Note: With all types of NTLM authentication, credentials are not passed via clear text although NTLMv1 has well-documented vulnerabilities.  The ISR is NTLMv2 capable although by default, older versions of Windows may negotiate via NTLMv1.  This behavior is dependent upon Active Directory authentication policies. Transparent NTLM NTLM authentication in which a user is logged into their workstation with domain credentials and those credentials are passed transparently by the browser to the IOS router which then performs an LDAP query to validate the user’s credentials.  This is generally the most desired form of authentication for this feature. Basic Authentication (via HTTP in clear text) This form of authentication is typically used as a fallback mechanism when NTLM authentication fails or is not possible for clients such as Macs, Linux-based devices, or mobile devices.  In this scenario, if HTTP Secure server is not enabled then these credentials are passed via HTTP in clear text (very insecure). Passive NTLM Passive NTLM authentication requests credentials from users but does not actually authenticate the user against the domain controller.  While this can avoid LDAP-related problems where the queries are failing against the DC, it also introduces a security risk.  If transparent authentication fails or is not possible then the users are prompted for credentials, however the user can enter any credentials they choose and those will be passed on to the CWS tower.  As a result, policies may not be applied appropriately.  For example, User A could use Firefox (which by default does not allow transparent NTLM without additional configuration) and enter the username of User B with any password and the policies for user B would be applied to user A.  The risk exposure can be mitigated if users are all forced to use browsers that support transparent NTLM authentication but in most cases the use of passive authentication is not recommended. Complete Message Sequence for Active NTLM Authentication: Browser -->  ISR : GET / google.com Browser <--  ISR : 302 Page moved http://1.1.1.1/login.html Browser -->  ISR : GET /login.html 1.1.1.1 Browser <--  ISR : 401 Unauthorized..Authenticate using NTLM Browser -->  ISR : GET /login.html + NTLM Type-1 msg ISR     -->  AD  : LDAP Bind Request + NTLM Type-1 msg ISR copies the Type-1 msg from HTTP to the LDAP byte-by-byte without altering any data ISR     <--  AD  : LDAP Bind Response + NTLM Type-2 msg Browser <--  ISR : 401 Unauthorized + NTLM Type-2 msg The Type-2 msg is also copied from byte-by-byte from LDAP to HTTP. So, in the pcap it appears to originate from 1.1.1.1 but the actual content is from AD. Browser -->  ISR : GET /login.html + NTLM Type-3 msg ISR     -->  AD  : LDAP Bind Request + NTLM Type-3 msg ISR     <--  AD  : LDAP Bind response - Success Browser <--  ISR : 200OK + redirect to google.com When Active NTLM is configured the ISR does not interfere during NTLM exchange. However, if passive is configured then the ISR generates its own Type-2 msg. Troubleshooting Show commands show ip admission cache show ip admission status show ip admission statistics show ldap server all Debug commands debug ldap all debug ip admission detail  (very verbose) debug ip admission httpd debug ip http transaction debug aaa authentication debug aaa authorization Common Problems IP Admission is not Intercepting HTTP requests This is evident by the output of ‘show ip admission statistics’ does not show any HTTP request being intercepted: kklous-881#show ip admission statistics Webauth HTTPd statistics:   HTTPd process 1     Intercepted HTTP requests:                            0 … Possible Solutions: Check to verify that ‘ip http server’ is enabled. If the ISR's HTTP server is not enabled then IP admission will trigger but never actually intercept the HTTP session and therefore prompt for authentication.  In this situation, there will be no output in the ‘show ip admission cache’ command but you will see many recurrences of the following in the output of ‘debug ip admission detail’ *Jan 30 20:49:35.726: ip_admission_det:proceed with process path authentication *Jan 30 20:49:35.726: AUTH-PROXY auth_proxy_find_conn_info :          find srcaddr - 14.36.104.152, dstaddr - 23.73.127.139                  ip-srcaddr 14.38.104.1                  pak-srcaddr 14.36.104.152 Check to verify that the user’s IP address is not exempt from the ACL in the IP Admission Configuration Users see a “404 Not Found” error in their browsers when redirected for authentication Possible Solution: Ensure that the name in the “ip admission virtual-ip 1.1.1.1 virtual-host ISR_PROXY” can successfully resolve with the client’s DNS server.  In this case, the client will perform a DNS query for “ISR_PROXY.kklous.cisco.com” since “kklous.cisco.com” is the fully-qualified domain name (FQDN) of the domain to which the workstation is joined.  If the DNS query fails, the client will send out a Link-Local Multicast Name Resolution (LLMNR) query followed by a NETBIOS query that is broadcast to the local subnet. If all of these resolution attempts fail then a “404 Not Found” or “Internet Explorer Cannot Display the webpage” error will be displayed in the browser.  See the following Wireshark screenshot for what this situation looks like in a packet capture: User is Prompted for Authentication but Authentication Always Fails This can be caused by various reasons but is usually related to LDAP configuration on the ISR or communication between the ISR and the LDAP server.  On the ISR, the symptom generally manifests itself by showing that users are stuck in the ‘INIT’ state once IP admission is triggered: kklous-881(config)#do sh ip admi cac Authentication Proxy Cache Client Name N/A, Client IP 14.36.104.152, Port 56674, timeout 60, Time Remaining 2, state INIT Common causes for this symptom include the following: Invalid username/password entered by the user for active authentication Invalid base-dn in LDAP configuration which results in searches returning no results Invalid bind authentication root-dn username or password which causing the LDAP bind to fail Communication between the ISR and LDAP server is failing (Verify that the LDAP server is listening on the specified TCP port for LDAP communication and that all firewalls between the two are allowing the traffic as well) Invalid search filter causes no results for LDAP search Troubleshooting LDAP The best way to determine the underlying reason why authentication is failing is to utilize LDAP debug commands on the ISR.  Keep in mind that debugs can be expensive and dangerous to run on an ISR if there is excessive output and can cause the router to hang and require a hard power cycle (this is especially true for the lower-end platforms). For troubleshooting, I recommend applying an ACL to the IP Admission rule to only subject a single test workstation on any production network to authentication.  That way you can enable debugs while minimizing any risk of negative impact to the router’s ability to pass traffic.  To date, I have never seen a router crash nor have problems passing traffic while running all IP Admission/LDAP-related debugs with only one user triggering the debug output. In the example below, we only subject the inside host with IP of 14.36.104.150 to authenticate while not requiring authentication for any other hosts: kklous-881(config)#ip access-list extended TEST_ADMISSION kklous-881(config-ext-nacl)#permit ip host 14.36.104.150 any kklous-881(config-ext-nacl)#deny ip any any kklous-881(config)#ip admission name SCANSAFE_ADMISSION ntlm list TEST_ADMISSION When troubleshooting LDAP-related problems.  It is helpful to understand the steps in which LDAP processes requests from the ISR. The High-Level Steps of LDAP Authentication Open connection to LDAP server on specified port (Default: TCP 389) Bind to LDAP server using the bind authenticate root-dn user and password Perform LDAP search for user attempting to authenticate using the base-dn and search-filters as defined in the LDAP configuration Obtain LDAP results from LDAP server and create an IP Admission cache entry for the user if successful or re-prompt for credentials in the event of an authentication failure. LDAP Debug Output Analysis These processes can be seen in the output of “debug ldap all”   Below is an example of LDAP debug output for an authentication that is failing due to an invalid base-dn.  Review the debug output and associated bolded comments that highlight the output that shows where the above processes may be failing: *Jan 30 20:51:50.818: LDAP: LDAP: Queuing AAA request 23 for processing *Jan 30 20:51:50.818: LDAP: Received queue event, new AAA request *Jan 30 20:51:50.818: LDAP: LDAP authentication request *Jan 30 20:51:50.818: LDAP: Username sanity check failed *Jan 30 20:51:50.818: LDAP: Invalid hash index 512, nothing to remove *Jan 30 20:51:50.818: LDAP: New LDAP request *Jan 30 20:51:50.818: LDAP: Attempting first  next available LDAP server *Jan 30 20:51:50.818: LDAP: Got next LDAP server :DC01 *Jan 30 20:51:50.818: LDAP: Free connection not available. Open a new one. *Jan 30 20:51:50.818: LDAP: Opening ldap connection ( 14.36.104.150, 389 )ldap_open    <-- This indicates that this is not a network layer issue since we have successfully opened the connection …. *Jan 30 20:51:50.822: LDAP: Root Bind on CN=Csco_Service,CN=Users,DC=kklous,DC=cisco,DC=com initiated. … *Jan 30 20:51:51.330: LDAP: Ldap Result Msg: SUCCESS, Result code =0 *Jan 30 20:51:51.330: LDAP: Root DN bind Successful on :CN=Csco_Service,CN=Users,DC=kklous,DC=cisco,DC=com     The 'Bind authenticate-dn' looks good here.  If the configuration was wrong for this then you'd see bind failures. … *Jan 30 20:51:51.846: LDAP: Received Bind Responseldap_parse_sasl_bind_result *Jan 30 20:51:51.846: LDAP: Ldap SASL Result Msg: SUCCESS, Result code =14 sasLres_code =14 *Jan 30 20:51:51.846: LDAP: SASL NTLM authentication do not require further tasks *Jan 30 20:51:51.846: LDAP: Next Task: All authentication task completed *Jan 30 20:51:51.846: LDAP: Transaction context removed from list [ldap reqid=14] *Jan 30 20:51:51.846: LDAP: * * AUTHENTICATION COMPLETED SUCCESSFULLY * * *Jan 30 20:51:51.846: LDAP: Notifying AAA: REQUEST CHALLENGED     <-- This indicates that all bind operations were successful and we are now proceeding to search for the actual user ….. *Jan 30 20:51:51.854: LDAP: SASL NTLM authentication done..Execute search   <-- LDAP Search debug output begins *Jan 30 20:51:51.854: LDAP: Next Task: Send search req *Jan 30 20:51:51.854: LDAP: Transaction context removed from list [ldap reqid=15] *Jan 30 20:51:51.854: LDAP: Dynamic map configured *Jan 30 20:51:51.854: LDAP: Dynamic map found for aaa type=username *Jan 30 20:51:51.854: LDAP: Ldap Search Req sent                     ld          2293572544                     base dn     dc=klous,dc=cisco,dc=com              <--notice the base-dn should be “kklous” and not “klous”                    scope       2                     filter      (&(objectclass=top)(sAMAccountName=testuser5))ldap_req_encode put_filter "(&(objectclass=top)(sAMAccountName=testuser5))" put_filter: AND put_filter_list "(objectclass=top)(sAMAccountName=testuser5)" put_filter "(objectclass=top)" put_filter: simple put_filter "(sAMAccountName=testuser5)" put_filter: simple Doing socket write *Jan 30 20:51:51.854: LDAP: lctx conn index = 2 … *Jan 30 20:51:52.374: LDAP: LDAP Messages to be processed: 1 *Jan 30 20:51:52.374: LDAP: LDAP Message type: 101 *Jan 30 20:51:52.374: LDAP: Got ldap transaction context from reqid 16ldap_parse_result *Jan 30 20:51:52.374: LDAP: resultCode:    10     (Referral) *Jan 30 20:51:52.374: LDAP: Received Search Response resultldap_parse_result ldap_err2string *Jan 30 20:51:52.374: LDAP: Ldap Result Msg: FAILED:Referral, Result code =10  <-- This incidates that the search returned no results (in this case due to an invalid base-dn)*Jan 30 20:51:52.374: LDAP: LDAP Search operation result : failedldap_msgfree *Jan 30 20:51:52.374: LDAP: Closing transaction and reporting error to AAA *Jan 30 20:51:52.374: LDAP: Transaction context removed from list [ldap reqid=16] *Jan 30 20:51:52.374: LDAP: Notifying AAA: REQUEST FAILED RFC 4511 - Lightweight Directory Access Protocol (LDAP): The Protocol Result code messages for LDAP and other LDAP protocol-related information can be found at the IETF website: http://tools.ietf.org/html/rfc4511#section-4.1.9 Known Caveats CSCum57928 - Scansafe needs a way to authenicate with HTTPS connection. CSCul22010 - High CPU Utilization when ip http secure server is enabled with NLTM CSCuc91949 - Scansafe IOS - Redirected HTTP site generating Tracebacks CSCum76723 - ENH: CWS-LDAP/NTLM auth should support multiple domains and forests ... View more