Cat 9000v and 802.1X / MAB support

[Arne Bier]

Hello,

CML (Cisco Modelling Lab) 2.5 announced a virtual Catalyst 9000 running IOS-XE 17.10 - I tested this out on my personal CML 2.6.1 and I am struggling to get 802.1X working.

I know Cat 9K virtual is still a beta release and the CML folks can't help me (they say the CML Team has nothing to do with the virtual Catalyst code). Perhaps someone on this community has used it and got Access Control (802.1X/MAB) working?

I provisioned the C9K via DNAC and also put all the usual config on that I do day in, day out for my customer networks. And TACACS+ and RADIUS is working fine. Device Tracking is also working fine.

When I connect an Ubuntu host (in CML) to the C9K Gig interface, it learns the MAC address, but it doesn't start any sessions.

I can ping the ubuntu host from the C9K, and vice-versa.

rnolab9K#show mac add int gi 1/0/2
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    5254.0015.e4dd    DYNAMIC     Gi1/0/2
Total Mac Addresses for this criterion: 1
rnolab9K#
rnolab9K#
rnolab9K#show access-session int gi 1/0/2 details 
No sessions match supplied criteria.

rnolab9K#
rnolab9K#show dot1x interface gigabitEthernet 1/0/2 details 
Dot1x Info for GigabitEthernet1/0/2
--------------------------------------------
PAE                       = AUTHENTICATOR
QuietPeriod               = 60
ServerTimeout             = 0
SuppTimeout               = 30
ReAuthMax                 = 3
MaxReq                    = 2
TxPeriod                  = 7

Dot1x Authenticator Client List Empty

Some useful config snippets

interface GigabitEthernet1/0/2
 description ubuntu
 switchport access vlan 10
 switchport mode access
 device-tracking attach-policy IPDT_POLICY
 authentication periodic
 authentication timer reauthenticate server
 access-session host-mode multi-host
 access-session control-direction in
 access-session port-control auto
 mab
 dot1x pae authenticator
 dot1x timeout tx-period 7
 dot1x max-reauth-req 3
 spanning-tree portfast
 service-policy type control subscriber PORT_AUTH_POLICY
end


aaa new-model
aaa group server radius dnac-client-radius-group
aaa authentication dot1x default group dnac-client-radius-group
aaa authorization network default group dnac-client-radius-group 
aaa accounting update newinfo periodic 2880
aaa accounting identity default start-stop group dnac-client-radius-group

aaa group server radius dnac-client-radius-group
 server name dnac-radius_172.22.131.174
 server name dnac-radius_172.22.131.175
 ip vrf forwarding Mgmt-vrf
 ip radius source-interface GigabitEthernet0/0
!
aaa server radius dynamic-author
 client 172.22.131.174 server-key 7 0336521D031D2F495A5B495744
 client 172.22.131.175 server-key 7 00361A10014905031B731C1C5A
!
ip radius source-interface GigabitEthernet0/0 
!
radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server attribute 31 mac format ietf upper-case
radius-server attribute 31 send nas-port-detail mac-only
radius-server dead-criteria time 5 tries 3
radius-server deadtime 3
!
dot1x system-auth-control
dot1x critical eapol
!
radius server dnac-radius_172.22.131.174
 address ipv4 172.22.131.174 auth-port 1812 acct-port 1813
 timeout 4
 retransmit 3
 automate-tester username dummy ignore-acct-port probe-on
 pac key 7 00361A10014905031B731C1C5A
!
radius server dnac-radius_172.22.131.175
 address ipv4 172.22.131.175 auth-port 1812 acct-port 1813
 timeout 4
 retransmit 3
 automate-tester username dummy ignore-acct-port probe-on
 pac key 7 0336521D031D2F495A5B495744
!
policy-map type control subscriber PORT_AUTH_POLICY
 event session-started match-all
  10 class always do-until-failure
   10 authenticate using dot1x retries 2 retry-time 0 priority 10
 event authentication-failure match-first
  5 class DOT1X_FAILED do-until-failure
   10 terminate dot1x
   20 authenticate using mab priority 20
  10 class AAA_SVR_DOWN_UNAUTHD_HOST do-until-failure
   10 activate service-template DEFAULT_CRITICAL_AUTH_ACCESS
   20 activate service-template DEFAULT_CRITICAL_VOICE_TEMPLATE
   30 authorize
   40 pause reauthentication
  20 class AAA_SVR_DOWN_AUTHD_HOST do-until-failure
   10 pause reauthentication
   20 authorize
  30 class DOT1X_NO_RESP do-until-failure
   10 terminate dot1x
   20 authenticate using mab priority 20
  40 class MAB_FAILED do-until-failure
   10 terminate mab
   20 authentication-restart 60
  60 class always do-until-failure
   10 terminate dot1x
   20 terminate mab
   30 authentication-restart 60
 event aaa-available match-all
  10 class IN_CRITICAL_AUTH_CLOSED_MODE do-until-failure
   10 clear-session
  20 class NOT_IN_CRITICAL_AUTH_CLOSED_MODE do-until-failure
   10 resume reauthentication
 event agent-found match-all
  10 class always do-until-failure
   10 terminate mab
   20 authenticate using dot1x retries 2 retry-time 0 priority 10
 event inactivity-timeout match-all
  10 class always do-until-failure
   10 clear-session
 event authentication-success match-all
 event violation match-all
  10 class always do-until-failure
   10 restrict
 event authorization-failure match-all
  10 class AUTHC_SUCCESS-AUTHZ_FAIL do-until-failure
   10 authentication-restart 60

The code version - this is still the same version that shipped with CML 2.5 - Cisco has not updated this image in a year - which is not a good sign ... I hope it's not a dead duck already

Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 8     C9KV-Q200-8P       17.10.01prd7      CAT9K_IOSXE           INSTALL

[pieterh]

I see a mismatch here :
     aaa server radius dynamic-author
          client 172.22.131.174 server-key 7 0336521D031D2F495A5B495744
          client 172.22.131.175 server-key 7 00361A10014905031B731C1C5A
     !
     radius server dnac-radius_172.22.131.174
     ...

     pac key 7 00361A10014905031B731C1C5A
     !
     radius server dnac-radius_172.22.131.175
     ...
     pac key 7 0336521D031D2F495A5B495744
     !
can you correct and test ?

I also miss this config line (but may be only not included in the snippets)
     dot1x system-auth-control

[Arne Bier]

Hi @pieterh 

Thanks for the sanity check. I can confirm that

rnolab9K#show run | in dot1x system-auth-control
dot1x system-auth-control
rnolab9K#

Interesting about those hex strings. All the aaa config was pushed by DNAC (latest version). I have never looked closely at those long hex strings. I have never had an issue with any DNAC provisioned switches. But this is the first time I am using C9Kv. And since DNAC uses PAC, I generally leave it alone.

The IOS-XE RADIUS config was not broken - I could see the aaa servers were all Alive, and I got responses from a "test aaa" command on the CLI.

To eliminate your suspicion, I manually configured the radius dynamic author (by pasting the plain-text string (as shown in ISE) into the IOS-XE config. Now I have this:

aaa server radius dynamic-author
 client 172.22.131.174 server-key 7 122B0C01171902013E79747A60
 client 172.22.131.175 server-key 7 02340D4D0E140124581C594B56

And I did some old-skool RADIUS configuration (pasting in the key 0 "string" from ISE)

radius server dnac-radius_172.22.131.174
 address ipv4 172.22.131.174 auth-port 1812 acct-port 1813
 timeout 4
 retransmit 3
 automate-tester username dummy ignore-acct-port probe-on
 key 7 0336521D031D2F495A5B495744
!
radius server dnac-radius_172.22.131.175
 address ipv4 172.22.131.175 auth-port 1812 acct-port 1813
 timeout 4
 retransmit 3
 automate-tester username dummy ignore-acct-port probe-on
 key 7 097E471F1C1719171F5E547878
!

With those changes in place I wanted to report back if this made any difference. But now MAC learning suddenly stopped working.

I think this C9K is more broken than I realised. The Gig1/0/2 interface is up/up but now it has no MAC address entries at all. Without MAC learning, this is not a useful switch. I bounced the interface. No change. I rebooted the ubuntu. No difference.

When I bounced the CML link (the "cable" that links ubuntu to C9K), the C9K rebooted itself.  I did mention this is beta. But it's more like alpha.

I'll reboot the C9K and see if that helps. But I think we're barking up the wrong tree here (suspecting RADIUS).

[MHM Cisco World]

Use server-private instead of server name since the server and interface use as source is connect via VRF

MHM

[Arne Bier]

Hi @MHM Cisco World 

Thanks for the suggestion. I have never had an issue with the config as per DNAC. I am using Gig0/0 within Mgmt-vrf - and I have no issues with sending RADIUS traffic to ISE (as confirmed when using the IOS-XE "test aaa" command). 

I think the DNAC config below clearly states the correct intent:

aaa group server radius dnac-client-radius-group
 server name dnac-radius_172.22.131.174
 server name dnac-radius_172.22.131.175
 ip vrf forwarding Mgmt-vrf
 ip radius source-interface GigabitEthernet0/0

As I replied to @pieterh earlier, the C9K is now playing up - once I get it to learn MAC addresses again (lol) I will try your suggestion for completeness's sake.

[pieterh]

try to refresh the radius server configuration  from DNAC
hope this restores MAC learning 

[Arne Bier]

MAC learning is not related to RADIUS at all. It's a basic functionality of any switch, ever since switches were invented.

I bounced the gig1/0/2 - normally that would awaken an endpoint that went to sleep. But in this case, it didn't.

I then rebooted the ubuntu PC to be 100% sure to wake it up. But still no MAC address on the switch.

Finally rebooted the C9K and then all the MAC learning worked again. For a few minutes and then it's dead in the water.

 I will poke around more tomorrow.

I was hoping that there was someone else using Cat9K virtual who had a better story than mine

I can ping the Ubuntu - but still, no MAC address on the switch

rnolab9K#ping 10.0.0.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/22/23 ms
rnolab9K#
rnolab9K#show ip arp 
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.0.0.1                -   5254.0014.6753  ARPA   Vlan10
Internet  10.0.0.10               0   5254.0015.e4dd  ARPA   Vlan10
rnolab9K#show mac address-table int gi 1/0/2
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
rnolab9K#

And here is the ubuntu side

[Torbjørn · Accepted solution]

I have not had any success with MAB on the Cat9kv 17.10 either. The image is very buggy in it's current state... Cisco will hopefully release a new version soon.

[Arne Bier]

After rebooting the C9K I have MAC address learning working again. And RADIUS is also still working as before. But the switch is not triggering session management.

rnolab9K#show mac address-table interface gi 1/0/2   
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    5254.0001.a588    DYNAMIC     Gi1/0/2
Total Mac Addresses for this criterion: 1
rnolab9K#
rnolab9K#show access-session interface gi 1/0/2 details 
No sessions match supplied criteria.

rnolab9K#term mon 
rnolab9K#debug radius
Radius protocol debugging is on
Radius protocol brief debugging is off
Radius protocol verbose debugging is off
Radius packet hex dump debugging is off
Radius packet protocol debugging is on
Radius elog debugging debugging is off
Radius RADSEC debugging debugging is off
Radius packet retransmission debugging is off
Radius table debugging is on
Radius server fail-over debugging is off
rnolab9K#test aaa group dnac-client-radius-group test test new-code 
User rejected

rnolab9K#
*Jan  4 10:39:13.953: %PARSER-5-HIDDEN: Warning!!! ' test platform-aaa group server-group dnac-client-radius-group user-name test test new-code blocked count delay level profile rate users ' is a hidden command. Use of this command is not recommended/supported and will be removed in future.
*Jan  4 10:39:13.960: RADIUS/ENCODE(00000000):Orig. component type = Invalid
*Jan  4 10:39:13.960: RADIUS/ENCODE: Skip encoding 0 length AAA attribute formatted-clid
*Jan  4 10:39:13.960: RADIUS(00000000): Config NAS IP: 172.22.128.6
*Jan  4 10:39:13.960: vrfid: [65535]  ipv6 tableid : [0]
*Jan  4 10:39:13.960: idb is NULL
*Jan  4 10:39:13.960: RADIUS(00000000): Config NAS IPv6: ::
*Jan  4 10:39:13.960: RADIUS(00000000): sending
*Jan  4 10:39:13.960: RADIUS/DECODE(00000000): There is no General DB. Want server details may not be specified
*Jan  4 10:39:13.961: RADIUS(00000000): Send Access-Request to 172.22.131.174:1812 id 1645/4, len 56
RADIUS:  authenticator 35 29 3F 51 3F 50 82 37 - 54 A7 A5 25 C5 28 CE 1C
*Jan  4 10:39:13.961: RADIUS:  User-Password       [2]   18  *
*Jan  4 10:39:13.961: RADIUS:  User-Name           [1]   6   "test"
*Jan  4 10:39:13.961: RADIUS:  Service-Type        [6]   6   Login                     [1]
*Jan  4 10:39:13.961: RADIUS:  NAS-IP-Address      [4]   6   172.22.128.6              
*Jan  4 10:39:13.961: RADIUS(00000000): Sending a IPv4 Radius Packet
*Jan  4 10:39:13.961: RADIUS(00000000): Started 4 sec timeout
*Jan  4 10:39:14.036: RADIUS: Received from id 1645/4 172.22.131.174:1812, Access-Reject, len 20
RADIUS:  authenticator 3C F7 CD 6B 7B 2B 8C 08 - 25 E0 B2 10 CA 46 BC 7A
*Jan  4 10:39:14.036: RADIUS/DECODE(00000000): There is no General DB. Reply server details may not be recorded
*Jan  4 10:39:14.036: RADIUS(00000000): Received from id 1645/4
rnolab9K#

The release notes state that DOT1X is supported. But I reckon the SMD (Session Management Daemon) is not implemented - I can't debug DOT1X like I normally do

rnolab9K#set platform software trace smd switch active R0 dot1x-all debug
rnolab9K#set platform software trace smd switch active R0 radius-authen debug
rnolab9K#set platform software trace smd switch active R0 aaa-authen debug
rnolab9K#set platform software trace smd switch active R0 eap-all debug
rnolab9K#set platform software trace smd switch active R0 auth-mgr-all debug
rnolab9K#
rnolab9K#show platform software trace message smd switch active R0
                                      ^
% Invalid input detected at '^' marker.

rnolab9K#show platform software trace ?                           
  context  Show btrace global context
  counter  Show counter value of modules
  level    Show trace levels

[MHM Cisco World]

Friend can you ping from mgmt interface to server using vrf mgmt in ping command?

MHM

[Arne Bier]

I don't suspect any issues with IP routing or RADIUS .. or ISE.

rnolab9K#ping vrf Mgmt-vrf 172.22.131.174
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.22.131.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
rnolab9K#

[MHM Cisco World]

Sorry can you check again 

Ping using source interface 

Thanks 

MHM

[Arne Bier]

You're barking up the wrong tree with that theory - IP routing is not the issue. It works. And RADIUS also works - the issue is that the session management daemon does not respond as expected, and MAC learning is flaky

But here goes ... pinging the ISE server from switch

rnolab9K#show ip int br
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  unassigned      YES NVRAM  up                    up      
Vlan10                 10.0.0.1        YES NVRAM  up                    up      
GigabitEthernet0/0     172.22.128.6    YES NVRAM  up                    up      
GigabitEthernet1/0/1   unassigned      YES unset  up                    up      
GigabitEthernet1/0/2   unassigned      YES unset  up                    up      
GigabitEthernet1/0/3   unassigned      YES unset  up                    up      
GigabitEthernet1/0/4   unassigned      YES unset  up                    up      
GigabitEthernet1/0/5   unassigned      YES unset  up                    up      
GigabitEthernet1/0/6   unassigned      YES unset  up                    up      
GigabitEthernet1/0/7   unassigned      YES unset  up                    up      
GigabitEthernet1/0/8   unassigned      YES unset  up                    up      
rnolab9K#show ip route vrf Mgmt-vrf
Extended Host Mode is enabled

Routing Table: Mgmt-vrf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR
       & - replicated local route overrides by connected

Gateway of last resort is 172.22.128.1 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 172.22.128.1
      172.22.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.22.128.0/24 is directly connected, GigabitEthernet0/0
L        172.22.128.6/32 is directly connected, GigabitEthernet0/0
rnolab9K#
rnolab9K#ping vrf Mgmt-vrf 172.22.131.174 source gigabitEthernet 0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.22.131.174, timeout is 2 seconds:
Packet sent with a source address of 172.22.128.6 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
rnolab9K#

And pinging the ubuntu host - which currently doesn't work, because, as I keep repeating, MAC address learning is broken. It was working for a few minutes after rebooting the C9Kv. The ubuntu host is powered up and link is up/up - by comparison, the MAC address learning on a trusty vIOS_l2 switch works as expected, sing the same ubuntu host. Regardless, the attached host sends enough traffic to keep the MAC ageing from expiring the CAM entry on the switch.

rnolab9K#ping 10.0.0.10 source vlan 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.10, timeout is 2 seconds:
Packet sent with a source address of 10.0.0.1 
.....
Success rate is 0 percent (0/5)

rnolab9K#show mac address-table             
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    0180.c200.0021    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
   1    5254.0014.6754    STATIC      Vl1 
  10    5254.0014.6753    STATIC      Vl10 
Total Mac Addresses for this criterion: 23
rnolab9K#

[MHM Cisco World]

rnolab9K#
rnolab9K#ping vrf Mgmt-vrf 172.22.131.174 source gigabitEthernet 0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.22.131.174, timeout is 2 seconds:
Packet sent with a source address of 172.22.128.6 
!!!!!

that Good
NOW 
are the mac is same always 
i.e. after you reboot the SW are the ubuntu MAC is same ?
MHM

[andrew-rob182]

I was using the IOSvL2 node and running into bugs as well. I had to use the "authentication open" command to get any EAP messages to go through. Might want to just try that and see if that changes anything. Its pretty buggy even on the L2 image. I couldn't get web-redirect to work, I moved the same configs over to a physical 2960 test switch I have and works perfectly.