RADIUS authentication fails after failover on nexus 1000v

[tgusset]

Hi

We have a Nexus 1110-S HA pair. RADIUS authentication works fine, till we have a failover (initiated by power off or system switchover).

The failover works fine but RADIUS authentication is no longer possible.

debug radius aaa-request logs:

ee-nexus1# 2014 Feb 19 18:19:28.561755 radius: get_radius_server_info_from_group:

2014 Feb 19 18:19:28.562018 radius: is_intf_up_with_valid_ip(1305):Can't determine interface mgmt0 status(up/down).

2014 Feb 19 18:19:28.562248 radius: radius_update_request_state_for_server(1373):No if_index configured for group mgmt0 or no_ip assigned

2014 Feb 19 18:19:28.562484 radius: radius_update_request_state_for_server(1375):Trying to retrieve intf info frm other groups to which this server belogns to..

2014 Feb 19 18:19:28.562722 radius: is_intf_up_with_valid_ip(1305):Can't determine interface mgmt0 status(up/down).

2014 Feb 19 18:19:28.562948 radius: radius_update_request_state_for_server(1399):Trying global...

2014 Feb 19 18:19:28.563180 radius: radius_update_request_state_for_server(1410):global interface  will be used

2014 Feb 19 18:19:28.563406 radius: radius_update_request_state_for_server(1421):global intf is not configured or up with valid ip.

2014 Feb 19 18:19:28.563650 radius: radius_update_request_state_for_server(1433):Using if_index

2014 Feb 19 18:19:28.563944 radius: getaddrinfo  serv_port 1645

2014 Feb 19 18:19:29.567825 radius: radius_set_src_intf(1707):setsockopt success, using src-intf:for server: 152.88.114.12 for sock: 22 Error returned:0x0 errno string:Invalid argument

2014 Feb 19 18:19:34.618054 radius: radius_set_src_intf(1707):setsockopt success, using src-intf:for server: 152.88.114.12 for sock: 22 Error returned:0x0 errno string:No route to host

2014 Feb 19 18:19:39.668358 radius: radius_request_process: RADIUS server 152.88.114.12 failed to respond evenafter all retries

2014 Feb 19 18:19:39.668841 radius: radius_request_process_next_server: All RADIUS servers failed to respond after retries.

ping to 152.88.114.12 works fine.

Configuration:

version 4.2(1)SP1(6.2)

feature telnet

username admin password 5 *  role network-admin

username orion password 5 *  role network-operator

banner motd #Cisco VSA#

ssh key rsa 2048

ip domain-lookup

ip domain-lookup

radius-server host 152.88.114.12 key 7 * auth-port 1645 acct-port 1646 authentication accounting

aaa group server radius AD-authentication

    server 152.88.114.12

    source-interface mgmt0

hostname ee-nexus1

snmp-server user admin network-admin auth md5 * priv * localize

dkey

snmp-server user orion network-operator auth sha * priv * localizedkey

snmp-server user orion oriongroup

ntp server 172.17.72.1

aaa authentication login default group AD-authentication

vrf context management

  ip route 0.0.0.0/0 172.17.72.1

vlan 1,372-373

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

vdc ee-nexus1 id 1

  limit-resource vlan minimum 16 maximum 2049

  limit-resource monitor-session minimum 0 maximum 2

  limit-resource vrf minimum 16 maximum 8192

  limit-resource port-channel minimum 0 maximum 768

  limit-resource u4route-mem minimum 32 maximum 32

  limit-resource u6route-mem minimum 16 maximum 16

  limit-resource m4route-mem minimum 58 maximum 58

  limit-resource m6route-mem minimum 8 maximum 8

network-uplink type 1

interface GigabitEthernet1

interface GigabitEthernet2

interface GigabitEthernet3

interface GigabitEthernet4

interface GigabitEthernet5

interface GigabitEthernet6

interface PortChannel1

virtual-service-blade VSM1

  virtual-service-blade-type name VSM-1.3

  interface control vlan 373

  interface packet vlan 373

  ramsize 3072

  disksize 3

  numcpu 1

  cookie 1669967837

  no shutdown

interface VsbEthernet1/1

interface VsbEthernet1/2

interface VsbEthernet1/3

interface mgmt0

  ip address 172.17.72.10/24

interface control0

clock timezone MET 1 0

clock summer-time MEST 5 sunday march 02:00 5 sunday October 03:00 60

line console

boot kickstart bootflash:/nexus-1010-kickstart-mz.4.2.1.SP1.6.2.bin

boot system bootflash:/nexus-1010-mz.4.2.1.SP1.6.2.bin

boot kickstart bootflash:/nexus-1010-kickstart-mz.4.2.1.SP1.6.2.bin

boot system bootflash:/nexus-1010-mz.4.2.1.SP1.6.2.bin

svs-domain

  domain id 1

  control vlan 373

  management vlan 372

  svs mode L2

logging server 152.88.6.10

debug radius aaa-request-lowlevel logs:

ee-nexus1# 2014 Feb 19 18:36:37.617895 radius: fsrv_sdb_process_msg: vdc-id[1] mts_opc[8441][MTS_OPC_RADIUS_NEW_AAA_REQ] 0xbfffe590 0xb5cc2c60 375

2014 Feb 19 18:36:37.618235 radius: fsrv didnt consume 8441 opcode

2014 Feb 19 18:36:37.618470 radius: process_aaa_radius_request: calling radius_pap_authenticate with user gutadm,servergroup AD-authentication, hostname

2014 Feb 19 18:36:37.618711 radius: radius_pap_authenticate: received PAP authentication request for xxxxxx

2014 Feb 19 18:36:37.618961 radius: get_radius_server_group_info: entering...

2014 Feb 19 18:36:37.619238 radius: RADIUS_FREE 0x8197264: radius_server_group_info_free

2014 Feb 19 18:36:37.619524 radius: build_radius_packet: entering for user xxxxxx

2014 Feb 19 18:36:37.619819 radius: radius_pap_authenticate: built the PAP RADIUS request packet, now send to the servers one by one

2014 Feb 19 18:36:37.620047 radius: radius_request_process: event: FIRST_REQUEST, switch to first server

2014 Feb 19 18:36:37.620268 radius: radius_request_process_next_server:

2014 Feb 19 18:36:37.620489 radius: radius_request_process_next_server: looping thru servers in servergroup...

2014 Feb 19 18:36:37.620722 radius: get_radius_server_info_from_group:

2014 Feb 19 18:36:37.620945 radius: radius_update_request_state_for_server:

2014 Feb 19 18:36:37.621186 radius: is_intf_up_with_valid_ip(1305):Can't determine interface mgmt0 status(up/down).

2014 Feb 19 18:36:37.621414 radius: radius_update_request_state_for_server(1373):No if_index configured for group mgmt0 or no_ip assigned

2014 Feb 19 18:36:37.621653 radius: radius_update_request_state_for_server(1375):Trying to retrieve intf info frm other groups to which this server belogns to..

2014 Feb 19 18:36:37.621890 radius: is_intf_up_with_valid_ip(1305):Can't determine interface mgmt0 status(up/down).

2014 Feb 19 18:36:37.622116 radius: radius_update_request_state_for_server(1399):Trying global...

2014 Feb 19 18:36:37.622347 radius: radius_update_request_state_for_server(1410):global interface  will be used

2014 Feb 19 18:36:37.622588 radius: radius_update_request_state_for_server(1421):global intf is not configured or up with valid ip.

2014 Feb 19 18:36:37.622813 radius: radius_update_request_state_for_server(1433):Using if_index

2014 Feb 19 18:36:37.623080 radius: radius_get_ip_local_from_remote(314): Setting context id to 1

2014 Feb 19 18:36:37.623304 radius: radius_get_ip_local_from_remote(319): Getting source IP for host 152.88.114.12.

2014 Feb 19 18:36:37.623611 radius: radius_get_ip_local_from_remote(322): Returning the source IP 0.0.0.0 for host 152.88.114.12.

2014 Feb 19 18:36:37.623842 radius: radius_update_request_state_for_server(1480): Using IP 0.0.0.0 as NAS IP.

2014 Feb 19 18:36:37.624074 radius: radius_update_request_state_for_server(1492): Can not find the source IP for destination IP152.88.114.12 topopulate the NAS IP.

2014 Feb 19 18:36:37.624299 radius: getaddrinfo  serv_port 1645

2014 Feb 19 18:36:37.624541 radius: Entering : check_local_cache : Line : 978

2014 Feb 19 18:36:37.624765 radius: serv_id : 152.88.114.12:1645

2014 Feb 19 18:36:37.624991 radius: Entry Found

2014 Feb 19 18:36:37.625211 radius: Entering : get_res_back : Line : 487

2014 Feb 19 18:36:37.625437 radius: No of entries in server_procjob_data_t : 1

2014 Feb 19 18:36:37.625669 radius: Exiting: get_res_back , Line : 564

2014 Feb 19 18:36:37.625897 radius: Found in Local Cache

2014 Feb 19 18:36:37.626124 radius: returning sockfd 22 for host 152.88.114.12:1645

2014 Feb 19 18:36:37.626352 radius: radius_request_process_next_server: found a valid server 152.88.114.12

2014 Feb 19 18:36:37.626589 radius: radius_request_process : Address family AF_INET

2014 Feb 19 18:36:37.626811 radius: radius_request_process(2869): neither local or global source interface configured

2014 Feb 19 18:36:37.627038 radius: radius_request_process(2887): vrf not configured

2014 Feb 19 18:36:37.627258 radius: radius_request_process(2888): Setting context id to 1

2014 Feb 19 18:36:37.627484 radius: sending 58 bytes to 152.88.114.12 port 1645 vrf

2014 Feb 19 18:36:37.633409 radius: radius_set_src_intf(1707):setsockopt success, using src-intf:for server: 152.88.114.12 for sock: 22 Error returned:0x0 errno string:Success

2014 Feb 19 18:36:37.633964 radius: radius_pap_authenticate: exiting

2014 Feb 19 18:36:37.634200 radius: process_aaa_radius_request: returning TRUE...

2014 Feb 19 18:36:42.680059 radius: radius_request_process : Address family AF_INET

2014 Feb 19 18:36:42.680283 radius: radius_request_process(2869): neither local or global source interface configured

2014 Feb 19 18:36:42.680506 radius: radius_request_process(2887): vrf not configured

2014 Feb 19 18:36:42.680727 radius: radius_request_process(2888): Setting context id to 1

2014 Feb 19 18:36:42.680958 radius: sending 58 bytes to 152.88.114.12 port 1645 vrf

2014 Feb 19 18:36:42.681529 radius: radius_set_src_intf(1707):setsockopt success, using src-intf:for server: 152.88.114.12 for sock: 22 Error returned:0x0 errno string:No route to host

2014 Feb 19 18:36:47.730761 radius: radius_request_process: RADIUS server 152.88.114.12 failed to respond evenafter all retries

2014 Feb 19 18:36:47.730994 radius: trying out next server

2014 Feb 19 18:36:47.731230 radius: radius_request_process_next_server:

2014 Feb 19 18:36:47.731452 radius: radius_request_process_next_server: looping thru servers in servergroup...

2014 Feb 19 18:36:47.731896 radius: radius_request_process_next_server: All RADIUS servers failed to respond after retries.

2014 Feb 19 18:36:47.732117 radius: pap_auth_info_unavail_func: entering for aaa session 0

2014 Feb 19 18:36:47.732348 radius: pap_reply: entering for aaa session: 0

2014 Feb 19 18:36:47.732590 radius: send_aaa_radius_resp_mts: entering for aaa session 0

2014 Feb 19 18:36:47.736242 radius: send_aaa_radius_resp_mts: exiting for aaa session 0

2014 Feb 19 18:36:47.736474 radius: pap_reply: exiting for aaa session: 0

2014 Feb 19 18:36:47.736696 radius: pap_auth_info_unavail_func: exiting for aaa session 0

2014 Feb 19 18:36:47.736920 radius: RADIUS_FREE 0x819a8c4: free_radius_state

2014 Feb 19 18:36:47.737145 radius: RADIUS_FREE 0x8194394: free_radius_state

Any idea what's wrong?

Thanks in advance

Thomas

[tgusset]

I found a solution

in

aaa group server radius AD-authentication

    server 152.88.114.12

    source-interface mgmt0

we should replace source-interface mgmt0 with use-vrf management