IEEE802.1x failing on switches with routing enabled

Jim Blake · ‎10-30-2012

I have been deploying IEEE802.1x to protect the ports on my network. This works well, when the protected ports are on a L2 access-only switch, that is to say, a switch that does not carry any SVIs or do any interVLAN routing. If I attempt to run IEEE802.1x ports on a switch that is both access and core, so has access ports *and* VLAN SVIs/handles routing between VLANs (Typical small site) then as soon as a user port attempts to authenticate a device, the switch throws up a "Server dead" notification in the log and dot1x fails.

I can't believe Cisco missed this during testing, and I don't want to have to make all my single switch sites into dual switch sites (one switch for access VLANs and IEEE802.1x, the other switch for SVIs and inter-VLAN routing). Has anyone found a way round this issue?

Thanks for any help

Jim

kcnajaf · ‎10-30-2012

Hi Jim,

Could you paste the configuration here.

If im not wrong I think what would be happening here is that switch is using a different SVI address (which does not have reachability to your athentication server).

You can specifically tell the switch to use which interface while carrying 802.1x authetocation using below command. Assuming you are using Radius authentication you could try below command

ip radius source-interface vlan XX (where XX is the interface which has reachability to your radius server)

Hope i have not misunderstood your problem :-)

Regards

Najaf

Jim Blake · ‎10-31-2012

Hi Najaf,

The config on my test rig is a bit messy, but I can tell you the following:

1) The switch is on 192.168.123.57

2) The RADIUS server is on 192.168.123.1

3) The switch can ping the server, the server can ping the switch

4) The RADIUS server has correct client and switch configs

5) When the switch tries to authenticate a device on a port, I see in wireshark an authentication request packet come in to the server from the switch, on port 1812, the RADIUS port, with the device ID (MAC address, as its a phone)

6) IMMEDIATELY after the authentication request comes in, I get an ICMP Destination unreachable (Port unreachable) notification in Wireshark, from the server to the switch.

The response coming off the server is a puzzle, the server can ping the switch and vice versa, but the server is telling the switch, from whom it just recieved a request, that it dosen't know how to get back to it...if I am interpreting the trace correctly...

Let me know if you still want the config, and I will sanitise it....

Thanks for your help

Jim

kcnajaf · ‎10-31-2012

Hi Jim,

Could you confirm couple of things?

1) Is your switch and RADIUS server on the same subnet (you have mentioned the ip address as 192.168.123.57 and 1)

2) Do you have any firewall between switch and RADIUS server?

3) On the Radius authentication logs are you able to see passed or failed response as you have mentioned request is coming to Radius server.

4) Could you paste your switch config here?

Regards

Najaf

nspasov · ‎10-31-2012

This thread is intriguing me as well and I would like to see the actual switch config as well. Along with that can you also provide:

1. Model and version of code on the switch

2. Output from "show aaa servers" before and after you attempt to authenticate a client

Tarik Admani · ‎11-01-2012

Hi

Can you check the source interface that is being used for radius requests. Also a debug radius authentication should let you know which interface is sourcing the packet.

Sent from Cisco Technical Support Android App

Jim Blake · ‎11-01-2012

Gentlemen

I appreciate all your interest and will reply with more facts as soon as I am able, but right now I have just taken my wife to the maternity ward, so responses other than cursory ones like this will be somewhat delayed...I'm sure you understand, and I hope you stay interested!

Thanks

Jim

kcnajaf · ‎11-01-2012

Hi Jim,

Expecting a good news soon then :-)

Sent from Cisco Technical Support iPhone App

Jim Blake · ‎11-06-2012

OK, Its time to see if I can answer some Questions:

Note, all the above relates to an attempt to authenticate (via MAB) a Cisco 7975 IP Phone, which was set up for IEEE802.1x, not CDP, authentication. MAC address f47f.35a3.cd60

1) Is your switch and RADIUS server on the same subnet (you have mentioned the ip address as 192.168.123.57 and 1)

Response: The switch is using VLAN2 as a common VLAN for switch SVI, Sever and the device to be authenticated. The server is 192.168.123.1/24, the switch is 192.168.123.57/24, and the device to be authenticated should pick up an address by DHCP from the same server (.1) which is also running as DHCP server (I tried it elsewhere with a different DHCP server, but it made no appreciable difference.)

2) Do you have any firewall between switch and RADIUS server?

Response: No, all elements of the test are on the same VLAN

3) On the Radius authentication logs are you able to see passed or failed response as you have mentioned request is coming to Radius server.

Response: I am getting nothing in the RADIUS logs. However, I am using FreeRADIUS, and independant testing using radtest shows the server to be operating correctly.

4) Could you paste your switch config here?

Response: Here's a sanitized version:

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Dot1xTest

!

boot-start-marker

boot-end-marker

!

aaa new-model

!

aaa authentication dot1x default group radius

aaa authorization network default group radius

!

system mtu routing 1500

!

vtp mode transparent

!

interface GigabitEthernet0/1

description #### client device to be authenticated ####

switchport access vlan 2

switchport mode access

authentication event fail action authorize vlan 9

authentication event no-response action authorize vlan 9

authentication host-mode multi-auth

authentication port-control auto

authentication periodic

authentication timer reauthenticate 28800

authentication violation protect

mab

dot1x pae authenticator

dot1x timeout quiet-period 600

dot1x timeout server-timeout 40

dot1x timeout tx-period 2

no shut

!

interface GigabitEthernet0/2

shutdown

!

interface GigabitEthernet0/3

shutdown

!

interface GigabitEthernet0/4

shutdown

!

interface GigabitEthernet0/5

shutdown

!

interface GigabitEthernet0/6

shutdown

!

interface GigabitEthernet0/7

shutdown

!

interface GigabitEthernet0/8

description #### link to RADIUS Server ####

switchport access vlan 2

switchport mode access

!

interface GigabitEthernet0/9

shutdown

!

interface GigabitEthernet0/10

shutdown

!

interface Vlan1

no ip address

!

ip classless

!

vlan 2

name common_VLAN

!

vlan 9

name dumping_VLAN

!

interface Vlan2

description #### SVI for Common VLAN ####

ip address 192.168.123.57 255.255.255.0

!

ip default-gateway 192.168.123.1

no ip http server

no ip http secure-server

!

ip sla enable reaction-alerts

!

radius-server host 192.168.123.1 auth-port 1812 acct-port 1813 key testing123

radius-server retransmit 1

radius-server deadtime 1

ip radius source-interface vlan 2

!

line con 0

line vty 5 15

!

end

5). Model and version of code on the switch

Dot1xTest#show hard

Cisco IOS Software, C3560C Software (C3560c405ex-UNIVERSALK9-M), Version 12.2(55)EX2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 18-May-11 15:35 by prod_rel_team

Image text-base: 0x00003000, data-base: 0x02800000

ROM: Bootstrap program is C3560C boot loader

BOOTLDR: C3560C Boot Loader (C3560C-HBOOT-M) Version 12.2(55r)EX11, RELEASE SOFTWARE (fc1)

Dot1xTest uptime is 31 minutes

System returned to ROM by power-on

System image file is "flash:/c3560c405ex-universalk9-mz.122-55.EX2/c3560c405ex-universalk9-mz.122-55.EX2.bin"

Dot1xTest#show inv

NAME: "1", DESCR: "WS-C3560CG-8PC-S"

PID: WS-C3560CG-8PC-S , VID: V02 , SN: FOC1619Y0J2

6). Output from "show aaa servers" before and after you attempt to authenticate a client

Response: Here it is:

Dot1xTest#show aaa servers

RADIUS: id 1, priority 1, host 192.168.123.1, auth-port 1812, acct-port 1813

State: current UP, duration 592s, previous duration 0s

Dead: total time 0s, count 0

Quarantined: No

Authen: request 0, timeouts 0

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 0

Author: request 0, timeouts 0

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 0

Account: request 0, timeouts 0

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 0

Elapsed time since counters last cleared: 9m

Dot1xTest#

*Mar 1 00:11:51.658: %ILPOWER-7-DETECT: Interface Gi0/1: Power Device detected: IEEE PD

*Mar 1 00:11:52.182: %ILPOWER-5-POWER_GRANTED: Interface Gi0/1: Power granted

*Mar 1 00:11:59.774: %AUTHMGR-5-START: Starting 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:11:59.810: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Mar 1 00:12:00.812: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Mar 1 00:12:10.265: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:12:10.265: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:12:10.265: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

Dot1xTest#

Dot1xTest#show aaa servers

RADIUS: id 1, priority 1, host 192.168.123.1, auth-port 1812, acct-port 1813

State: current UP, duration 733s, previous duration 0s

Dead: total time 0s, count 0

Quarantined: No

Authen: request 2, timeouts 2

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 1

Author: request 0, timeouts 0

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 0

Account: request 0, timeouts 0

Response: unexpected 0, server error 0, incorrect 0, time 0ms

Transaction: success 0, failure 0

Elapsed time since counters last cleared: 12m

Dot1xTest#

7). Can you check the source interface that is being used for radius requests. Also a debug radius authentication should let you know which interface is sourcing the packet.

Response: I am forcing the RADIUS request to come from VLAN2 with the command

" ip radius source-interface vlan 2"

8). Log Output during the tests shown below:

Log Buffer (4096 bytes):

.233: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:33:53.975: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:33:53.975: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:33:53.975: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:35:04.109: %RADIUS-4-RADIUS_DEAD: RADIUS server 192.168.123.1:1812,1813 is not responding.

*Mar 1 00:35:14.495: %RADIUS-3-ALLDEADSERVER: Group radius: No active radius servers found. Id 20.

*Mar 1 00:35:14.495: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:35:14.495: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:35:14.495: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:36:04.114: %RADIUS-6-SERVERALIVE: Group radius: Radius server 192.168.123.1:1812,1813 is responding again (previously dead).

*Mar 1 00:36:04.114: %RADIUS-4-RADIUS_ALIVE: RADIUS server 192.168.123.1:1812,1813 is being marked alive.

*Mar 1 00:36:25.242: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:36:25.242: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:36:25.242: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:37:36.441: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:37:36.441: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:37:36.441: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:38:47.267: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:38:47.267: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:38:47.267: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:39:58.350: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:39:58.350: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:39:58.350: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B3900000000000AF3FF

*Mar 1 00:41:05.485: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi0/1: PD removed

*Mar 1 00:41:06.507: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

*Mar 1 00:41:07.509: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

*Mar 1 00:41:08.573: %RADIUS-4-RADIUS_DEAD: RADIUS server 192.168.123.1:1812,1813 is not responding.

*Mar 1 00:41:18.749: %RADIUS-3-ALLDEADSERVER: Group radius: No active radius servers found. Id 25.

*Mar 1 00:42:08.578: %RADIUS-6-SERVERALIVE: Group radius: Radius server 192.168.123.1:1812,1813 is responding again (previously dead).

*Mar 1 00:42:08.578: %RADIUS-4-RADIUS_ALIVE: RADIUS server 192.168.123.1:1812,1813 is being marked alive.

Dot1xTest#

9). Aside from all that, Holly Emma arrived at 02:52 on Sunday morning (Why do they do that ??? I need my sleep!)

Once again, thanks for your interest Guys

Best Regards

Jim

kcnajaf · ‎11-06-2012

Congrads Jim....

The configuration looks fine.

Are you sure that your radius server is configured to to listen port 1812? My understanding is that few of the radius servers still use port 1645 for radius authetication.

If you are not sure about this please try changing the configuration of switch as below.

no radius-server host 192.168.123.1 auth-port 1812 acct-port 1813 key testing123

radius-server host 192.168.123.1 auth-port 1645 acct-port 1646 key testing123

Also enable debug aaa authetication and test a radius authetication with below command and paste the output.

test aaa group radius

Regards

Najaf

nspasov · ‎11-07-2012

First off, congratulations!

It is very interesting that the AAA server is showing as "UP" but then it goes down as soon as you attempt to perform an authentication. It is starting to smell like a bug

I think testing a username/password (suggested by Najaf) is a good idea. Also, can you add "authentication open" and then try to authenticate the device again. It will be interesting to know if you get any hits in ISE from either one of the two.

Jim Blake · ‎11-08-2012

Hi Again Guys,

once again, I'll group the answers so you each have info to work with:

1). "Are you sure that your radius server is configured to to listen port 1812?"

Response: Yes, certain. I'm running FreeRadius and I can set it to whatever I need, but the default, which I'm running, is 1812. Also, when I run FreeRadius from the command line (#freeradius -X) I get the following output at startup:

"Listening on authentication address * port 1812

Listening on accounting address * port 1813

Listening on authentication address 127.0.0.1 port 18120 as server inner-tunnel

Listening on proxy address * port 1814

Ready to process requests."

2). When I tried the Debug, I got the following:

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0

% Incomplete command.

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0 ?

legacy Code path through legacy

new-code Code path through new-code

port Port send as part of request

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0 new

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0 new-code

*Mar 1 00:24:56.962: AAA/AUTHEN/8021X (00000002): Pick method list 'default'

*Mar 1 00:24:57.041: AAA/AUTHEN/LOGIN (00000000): Pick method list 'Permanent Local' User rejected

Dot1xTest#

*Mar 1 00:25:06.688: %MAB-5-FAIL: Authentication failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B390000000000021FAE

*Mar 1 00:25:06.688: %AUTHMGR-7-RESULT: Authentication result 'server dead' from 'mab' for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B390000000000021FAE

*Mar 1 00:25:06.688: %AUTHMGR-5-FAIL: Authorization failed for client (f47f.35a3.cd60) on Interface Gi0/1 AuditSessionID C0A87B390000000000021FAE

Dot1xTest#

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0 leg

Dot1xTest#test aaa group radius f47f35a3c6d0 f47f35a3c6d0 legacy

Attempting authentication test to server-group radius using radius

*Mar 1 00:25:25.552: AAA: parse name= idb type=-1 tty=-1

*Mar 1 00:25:25.552: AAA/MEMORY: create_user (0x44C3328) user='f47f35a3c6d0' ruser='NULL' ds0=0 port='' rem_addr='NULL' authen_type=ASCII service=LOGIN priv=1 initial_task_id='0', vrf= (id=0)No authoritative response from any server.

Dot1xTest#

*Mar 1 00:25:35.786: %RADIUS-4-RADIUS_DEAD: RADIUS server 192.168.123.1:1812,1813 is not responding.

*Mar 1 00:25:35.786: AAA/MEMORY: free_user (0x44C3328) user='f47f35a3c6d0' ruser='NULL' port='' rem_addr='NULL' authen_type=ASCII service=LOGIN priv=1 vrf= (id=0)

That all looks a bit inconclusive. If I try it with "authentication open" the symptoms dop not change: the Radius server remains pingable throughout, but is noted as "dead", coming back to life occasionally, but going "dead" again as soon as the switch tries to use it...

3). "It is starting to smell like a bug"...That was my feeling, but I just find it difficult to believe that Cisco would leave such an obvious bug in place, over many versions of IOS. This is a common issue, it happens on 2960s, 3560s, 3750s, etc

Over to you!

Thanks for your continued interest and patience with my patchy response times!

Cheers

Jim

nspasov · ‎11-08-2012

I just realized that I need to read the posts better...this whole time I thought you are using Cisco ISE I just tested this in my lab on a 3750 that has IP routing enabled with 2 SVIs and EIGRP between the 3750 and the core (4500) and things are working normal. The catch is I am using ISE in my lab

Also, did you get any failed authentications in your Radius server when you tried the "test aaa..." command?

Jim Blake · ‎11-13-2012

Hi Guys,

Having an "interesting" time here, with one or two medical problems distracting me from the 802.1x problem. Will get back to you as soon as I can..

JB