Trouble with C9500 vrf and MAB

[Andi1075]

ello Community,

we have some issues concerning VRF and MAB that we don't understand.

Our Platform is a C9500, network essential smart license.
We use the Mgmt-interface gi0/0 for security reasons and a central radius server for authentication, which works just fine. The only issue that we were not able to solve is MAB in order with the Mgmt-vrf.
while the login (aaa) proccess works fine with VRF , MAB always try to send packets thro the data plan even tho configured differnetly.

Is there a know bug in the IOS XE ? ( we are currently on 17.9.3)

Related Config:

vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
aaa new-model
!
!
aaa group server radius freerad
  server name ONE
  server name TWO
  ip vrf forwarding Mgmt-vrf
  ip radius source-interface GigabitEthernet0/0
!
aaa authentication login default local group freerad
aaa authentication dot1x default group freerad
aaa authorization exec default local group freerad
aaa authorization network default group freerad
!
license boot level network-essentials addon dna-essentials
dot1x system-auth-control

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address x.x.x.x 255.255.254.0
negotiation auto
!
!
interface TwentyFiveGigE1/0/4
switchport
switchport access vlan 999
switchport mode access
switchport nonegotiate
no cdp enable
authentication port-control auto
authentication periodic
mab
storm-control broadcast level 75.00
storm-control multicast level 75.00
storm-control action shutdown
spanning-tree portfast edge
!
radius server ONE
address ipv4 192.168.0.131 auth-port 1812 acct-port 1813
retransmit 3
key 7 12341234
!
radius server TWO
address ipv4 192.168.0.132 auth-port 1812 acct-port 1813
retransmit 3
key 7 12341234

!

ip radius source-interface GigabitEthernet0/0 vrf Mgmt-vrf

A vrf route is set to the router and login on VTY works perfectly fine, so does ntp. Just no mab, its like the MAB process doesn't use the VRF interface even tho configured. Erything works fine if we add a SVI and use the SVI instead of the Management Interface - but due to sercurity reasons this is not an option.

Thanks for any advice.
Andy

[Jens Albrecht]

Hello @Andi1075,

the management port with the vrf mgmt-vrf is dedicated for OOB management. So management-plane protocols like SSH can use aaa authentication via the mgmt-vrf without problems.

However, protocols like 802.1X or MAB are data-plane protocols and therefore not allowed to use the mgmt-vrf. Instead the data-plane protocols use the global vrf by default as you noticed.

I understand your security concerns and a common solution is to create a dedicated vrf for your authentication traffic.

So you create a new vrf e.g. VRF-RADIUS, place a SVI or a routed interface in this new vrf and assign this new vrf and source-interface to your radius-server group freerad.

Yes, the separation using VRFs is not as perfect as using the OOB port but it works.

HTH!

[Andi1075]

@Jens Albrecht Thank you very much for helping me out. Was there any change to this after 16.12 ? I am asking because it works without a flaw on 3650s, 3850s  that are running on IOS XE 16.12.3.

Thanks again !

[Jens Albrecht]

Hello @Andi1075,

I never tried this on the old 3650/3850s but I assume that this is more an issue of the new platform than software related.

Over the past years Cisco has done quite a lot to improve the device hardening and restricting data-plane protocols from accessing the mgmt-vrf is just a logical step following this strategy.

For high-security environments you can configure 2 separate radius-server groups if needed. One is used for SSH only, while the other group is used for 802.1X/MAB. This also works well but adds a bit of complexity, of course.

HTH!

[Andi1075]

Hello @Jens Albrecht,

well this would solve this issue but kind of blow's the use of it since I would have to add another pair of fiber per switch to the already 2 fiber pairs ( 1 Mgmt, 1  data trunk) . Thats way to much overhead. Also setting up a additional Radius in the data-plane is not going to happen for us since the data-plane is not to be altered server wise.
It still is strange that NTP is working over the Mgmt-vrf but radius isn't-... I disagree with Cisco that radius is no management protocol. In my eyes it shouldn't be in the data plan, but who am I to judge.

Thanks again ! 

[noahallenuk29]

Running MAB (MAC Authentication Bypass) on a C9500 with VRFs can definitely get tricky, especially when it comes to RADIUS traffic routing and source interface selection. A few key things to check:

VRF-aware AAA configuration – Make sure your RADIUS server is reachable via the correct VRF. Use the ip radius source-interface and ip vrf forwarding commands appropriately.

Correct interface sourcing – If the authentication requests aren’t leaving from the interface in the correct VRF, try explicitly binding your RADIUS traffic using:

bash
Copy
Edit
radius-server source-interface <interface-name>
And ensure that interface is in the correct VRF context.

Control-plane policing or ACLs – Sometimes VRF misbehavior during MAB is due to CoPP policies or misconfigured ACLs that drop RADIUS or DHCP packets silently.

Debug tools – Use:

bash
Copy
Edit
debug radius
debug mab all
Alongside show authentication sessions and show vrf to validate traffic flow.

Bonus tip: If you’re mixing dot1x and MAB, check your fallback configuration. Misalignments between authentication order and method can silently fail MAB after dot1x timeout.

Let us know your IOS-XE version and topology if you'd like more tailored advice!

[Jens Albrecht]

Hello @Andi1075,

there is one thing that you can try but I never tested it for this kind of scenario.

You could set up static route leaking between the Mgmt-vrf and the Global vrf and see whether this solves your problem.
It is a common technique to route between your own VRFs and the Global vrf but I never tested this with the build-in Mgmt-vrf. Nevertheless, it might be worth a try and please let me know the result in case you decide to test this.

Regards, Jens

[Andi1075]

Hi @Jens Albrecht, adding a route from the data-plane towards the Mgmt-Plane should (according to Cisco) create a loop and crash the switch (as far as I remember). So no, great idea but that wont work I guess  

@noahallenuk29 the radius server interface is configured correlty, ssh over Mgmt works perfectly fine, which wouldnt be the case if it wasn't.
I could find some CCP policies , but I have to admit that I never altered them ... auth dot1x was permitted as far as I could identify. 
Is there a way to permit radius too ? 

Thanks ! 

[Andi1075]

... not sure but it seems a lot like this bug :

CSCvv81868 : Bug Search Tool

apart from that we are running 17.9.3 - is the bug back ? 

[Jens Albrecht]

Yes, the description sounds indeed very similar. It does not happen very often but I saw a few bugs coming back in later releases.

Open a TAC case and ask them directly to confirm this.