Hello everyone ,
I´ve implemented 2 Cisco ISE v1.1 in HA to run MAB and 802.x Authentication / Authorization. Using Local ISE DB and Active Directory as an External Identity Source for wireless and wired users and devices. This was working fine 2 weeks ago after finishing installation.
My NAD devices are a Core SW 6500 for wired users (there are no access SW, just the Core for the whole network, its a small office) and a WLC 2405 for Wireless Users.
Here is the network topology:
DNSs are fully resolvable forward and reverse zone and ISEs, AD, WLC and SW Core are synched with the same NTP server.
As I mentioned Authentication and Authorization were working fine. Two weekends ago there was an electrical outage in the office. When the ISEs servers came up, the trust relationship between AD and ISEs was broken and so was HA replication. I did some troubleshoot to delete and install new certificates from AD into both ISEs and build again the HA configuration. I finally got the ISEs working fine again.
This last weekend, another electrical outage occurs in the office (client is working with a temporary plant and is already warned about electrical damages not covered by warranty) and the ISE servers came up in the same condition again, no trust relationship with AD (Domain Controller). So I fix this again by deleting and installing new certificates into ISE. The problem is that for some reason the NAD 6500 is not authenticating to the ISE. I´m receiving the following debug messages in the SW:
Sep 12 17:41:00.222: RADIUS(00000000): Request timed out
Sep 12 17:41:00.222: RADIUS: Retransmit to (172.16.3.5:1812,1813) for id 1645/165
Sep 12 17:41:00.222: RADIUS(00000000): Started 5 sec timeout
Sep 12 17:41:00.226: RADIUS: Received from id 1645/165 172.16.3.5:1812, Access-Reject, len 20
Sep 12 17:41:00.226: RADIUS: authenticator 00 D5 B6 0B C9 49 83 81 - 87 17 23 82 2B 6A CB C7
Sep 12 17:41:00.226: RADIUS: response-authenticator decrypt fail, pak len 20
Sep 12 17:41:00.226: RADIUS: packet dump: 03A5001400D5B60BC9498381871723822B6ACBC7
Sep 12 17:41:00.226: RADIUS: expected digest: BFAB772B5BA4B134F46E13A21F722317
Sep 12 17:41:00.226: RADIUS: response authen: 00D5B60BC9498381871723822B6ACBC7
Sep 12 17:41:00.226: RADIUS: request authen: 41EAE3A7DAEE6332CE646436F949C5A1
Sep 12 17:41:00.226: RADIUS: Response (165) failed decrypt
Sep 12 17:41:05.110: RADIUS(00000000): Request timed out
Sep 12 17:41:05.110: RADIUS: Retransmit to (172.16.3.5:1812,1813) for id 1645/165
Sep 12 17:41:05.110: RADIUS(00000000): Started 5 sec timeout
Sep 12 17:41:05.114: RADIUS: Received from id 1645/165 172.16.3.5:1812, Access-Reject, len 20
Sep 12 17:41:05.114: RADIUS: authenticator 00 D5 B6 0B C9 49 83 81 - 87 17 23 82 2B 6A CB C7
Sep 12 17:41:05.114: RADIUS: response-authenticator decrypt fail, pak len 20
Sep 12 17:41:05.114: RADIUS: packet dump: 03A5001400D5B60BC9498381871723822B6ACBC7
Sep 12 17:41:05.114: RADIUS: expected digest: BFAB772B5BA4B134F46E13A21F722317
Sep 12 17:41:05.114: RADIUS: response authen: 00D5B60BC9498381871723822B6ACBC7
Sep 12 17:41:05.114: RADIUS: request authen: 41EAE3A7DAEE6332CE646436F949C5A1
Sep 12 17:41:05.114: RADIUS: Response (165) failed decrypt
Sep 12 17:41:10.438: RADIUS(00000000): Request timed out
Sep 12 17:41:10.438: RADIUS: No response from (172.16.3.5:1812,1813) for id 1645/165
Sep 12 17:41:10.438: RADIUS/DECODE: parse response no app start; FAIL
Sep 12 17:41:10.438: RADIUS/DECODE: parse response; FAIL
Sep 12 17:41:13.682: %MAB-5-FAIL: Authentication failed for client (a44c.11ca.eadf) on Interface Gi1/29
Sep 12 17:41:13.682: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (a44c.11ca.eadf) on Interface Gi1/29
Sep 12 17:41:13.682: %AUTHMGR-5-FAIL: Authorization failed for client (a44c.11ca.eadf) on Interface Gi1/29
Sep 12 17:41:00.222: RADIUS(00000000): Request timed out
I have deleted and created again the 6500 NAD in the ISE, and configured againd the Radius-Key in the 6500 making sure they are exactly the same. But I keep receiving the same errors.
I have already reviewed the following links:
http://www.cisco.com/en/US/docs/wireless/access_point/12.3_7_JA/configuration/guide/s37err.html
http://www.cisco.com/en/US/docs/security/ise/1.1.1/user_guide/ise_logging.html#wp1061989
http://puck.nether.net/pipermail/cisco-nas/2004-May/000686.html
And the troubleshooting section from the Cisco Identity Services Engine User Guide, Release 1.0.4
Everything points me that the Radius Key between ISE and the 6500SW is wrong, but I´ve configured it again twice and typed it letter by letter slowly to avoid any typos.
ISE version: 1.1.0.665
ADE OS: 2
Active Directory: Windows 2008 R2 Standard
6500 SW Config:
Building configuration...
Current configuration : 65413 bytes
!
! Last configuration change at 12:22:42 MXVeran Tue Jul 31 2012 by ho1a
! NVRAM config last updated at 22:21:11 MXVeran Mon Jul 30 2012 by ho1a
!
version 15.0
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service compress-config
service counters max age 5
!
!
boot-start-marker
boot system flash bootdisk:
boot-end-marker
!
!
logging buffered 64000
enable secret 5 $1$QoxK$w6sZJ66pXDMLS1lGPp3KR.
!
username ho1a privilege 15 secret 5 $1$DYMo$O8BQi2u.emzdCFfNMxCTd.
username test-radius password 7 14141B180F0B7B7977
aaa new-model
!
!
aaa authentication login Tr3s41ia.2012 local
aaa authentication dot1x default group radius
aaa authorization exec default local
aaa authorization network default group radius
aaa authorization auth-proxy default group radius
aaa accounting update periodic 5
aaa accounting dot1x default start-stop group radius
aaa accounting system default start-stop group radius
!
!
aaa server radius dynamic-author
client 172.16.3.5 server-key 7 110A1016141D5A5E57
!
aaa session-id common
platform ip cef load-sharing ip-only
platform rate-limit layer2 port-security pkt 300 burst 10
clock timezone MXInv -6
clock summer-time MXVerano recurring
authentication critical recovery delay 1000
!
!
!
!
interface GigabitEthernet8/1
switchport
switchport access vlan 2
switchport mode access
ip access-group ACL_ISE_Default in
authentication host-mode multi-auth
authentication open
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
mab
dot1x pae authenticator
spanning-tree portfast edge
!
!
ip default-gateway 172.16.3.2
!
ip forward-protocol nd
ip http server
ip http secure-server
ip route 0.0.0.0 0.0.0.0 172.16.3.2
!
!
ip radius source-interface Vlan3 vrf default
logging origin-id ip
logging source-interface Vlan3
logging host 172.16.3.5 transport udp port 20514
!
snmp-server group Tr3s41ia.2012aes v3 priv
snmp-server group Tr3s41ia.2012md5 v3 auth
snmp-server community public RO
snmp-server community tresaliarw RW
snmp-server community tresaliaro RO
snmp-server trap-source Vlan3
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps memory bufferpeak
no snmp-server enable traps entity-sensor threshold
snmp-server enable traps cpu threshold
snmp-server enable traps vtp
snmp-server enable traps flash insertion removal
snmp-server enable traps mac-notification move change
snmp-server enable traps errdisable
snmp-server host 172.16.3.4 version 3 priv Tr3s41ia.2012aes
snmp-server host 172.16.3.4 version 3 auth Tr3s41ia.2012md5
snmp-server host 172.16.3.5 version 2c tresaliaro
!
radius-server attribute 6 on-for-login-auth
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server dead-criteria time 30 tries 3
radius-server host 172.16.3.5 auth-port 1812 acct-port 1813 test username test-radius key 7 104D000A061843595F
radius-server vsa send accounting
radius-server vsa send authentication
!
control-plane
service-policy input policy-default-autocopp
!
!
line con 0
logging synchronous
login authentication Tr3s41ia.2012
line aux 0
line vty 0 4
login authentication defaulTr3s41ia.2012
transport input ssh
line vty 5 1509
login authentication defaulTr3s41ia.2012
transport input ssh
!
ntp clock-period 17179836
ntp peer 172.16.4.9
no event manager policy Mandatory.go_switchbus.tcl type system
!
end
Additionaly, I´m getting the following screen when accesing the Stand-by server via https:
I´m thinking that there might be some problems with the CA Certificates installed on ISEs, or some corrupted data due to the 2 sudden restarts.
Any help, hint or direction will be really appreciated.
Thanks in advanced for your time. Best Regards.
Hello Tarik, thanks for your response,
I´ll go ahead and remove and configure again the complete radius configuration on the SW and let you know what happens, if this doesn´t work I´m thinking that re-installing the ISE server might be the solution. It´s was working fine after the fresh install.
I use the command "test aaa group radius username password new-code" to test SW communication to ISE and here is the debug output from the SW:
Sep 12 20:42:59.713: RADIUS/ENCODE(00000000):Orig. component type = INVALID
Sep 12 20:42:59.713: RADIUS(00000000): Config NAS IP: 172.16.3.1
Sep 12 20:42:59.713: RADIUS(00000000): sending
Sep 12 20:42:59.713: RADIUS(00000000): Send Access-Request to 172.16.3.5:1812 id 1645/93, len 56
Sep 12 20:42:59.713: RADIUS: authenticator 24 52 30 41 B7 06 74 CE - C7 4B 7B FF 87 88 F7 23
Sep 12 20:42:59.713: RADIUS: User-Password [2] 18 *
Sep 12 20:42:59.713: RADIUS: User-Name [1] 6 test
Sep 12 20:42:59.713: RADIUS: Service-Type [6] 6 Login [1]
Sep 12 20:42:59.713: RADIUS: NAS-IP-Address [4] 6 172.16.3.1
Sep 12 20:42:59.713: RADIUS(00000000): Started 5 sec timeout
Sep 12 20:43:14.485: RADIUS(00000000): Started 5 sec timeout
Sep 12 20:43:14.489: RADIUS: Received from id 1645/93 172.16.3.5:1812, Access-Reject, len 20
Sep 12 20:43:14.489: RADIUS: authenticator B2 89 18 4B F5 D8 D6 67 - 85 4D 1E C3 DE C9 06 85
Sep 12 20:43:14.489: RADIUS: response-authenticator decrypt fail, pak len 20
Sep 12 20:43:14.489: RADIUS: packet dump: 035D0014B289184BF5D8D667854D1EC3DEC90685
Sep 12 20:43:14.489: RADIUS: expected digest: EDB6C64ADA12BCD81CD21C3EF28CDB27
Sep 12 20:43:14.489: RADIUS: response authen: B289184BF5D8D667854D1EC3DEC90685
Sep 12 20:43:14.489: RADIUS: request authen: 24523041B70674CEC74B7BFF8788F723
Sep 12 20:43:14.489: RADIUS: Response (93) failed decryptUser rejected
And here are the results from the Operations/Authentications Tabe from ISE:
There are no other SWs in the network, just the Core. I cannot test Wireless Authentication since the AccessPoint Switchport is also controlled by ISE and is not Authenticated right now. I can Authenticate the Active Directory Users using NTRadPing tool as a test and its succesful. AD and 6500 SW are using the same Radius key to communicate with ISE. Here is the AD usert Authentication:
So I´ll proceed to re-configure the SW for Radius server and let you know if this is the solution.
Thanks in advanced for your time and comments.
