Cisco 9300 Stack Issues

[joeharb]

We have 2 9300's in a stack and are noticing a large amount of Total output drops on the interfaces.  Every up port seems to have output errors and even after a clear interfaces stats....but the output drops increase even with Minimum load.

We also have dot1x configured on the switch (same/similar config as used on other 4500's and 9300's...3650...) but nothing authenticates and we never see the switch send any packets to ISE.  I have removed all radius servers to be able to do a packet capture on ISE to the a specific one...I don't see anything.  The source interfaces are all defined and the proper vrf is specified.

Show aaa servers give the following:

RADIUS: id 1, priority 1, host 10.4.37.151, auth-port 1812, acct-port 1813
State: current UP, duration 4294967s, previous duration 0s
Dead: total time 0s, count 0
Platform State from SMD: current DEAD, duration 566s, previous duration 11s
SMD Platform Dead: total time 5065724s, count 8443
Platform State from WNCD: current UP, duration 0s, previous duration 0s
Platform Dead: total time 0s, count 0
Quarantined: No
Authen: request 88, timeouts 88, failover 0, retransmission 72
Response: accept 0, reject 0, challenge 0
Response: unexpected 0, server error 0, incorrect 0, time 0ms
Transaction: success 0, failure 16
Throttled: transaction 0, timeout 0, failure 0
Author: request 0, timeouts 0, failover 0, retransmission 0
Response: accept 0, reject 0, challenge 0
Response: unexpected 0, server error 0, incorrect 0, time 0ms
Transaction: success 0, failure 0
Throttled: transaction 0, timeout 0, failure 0
Account: request 0, timeouts 0, failover 0, retransmission 0
Request: start 0, interim 0, stop 0
Response: start 0, interim 0, stop 0
Response: unexpected 0, server error 0, incorrect 0, time 0ms
Transaction: success 0, failure 0
Throttled: transaction 0, timeout 0, failure 0
Elapsed time since counters last cleared: 44m
Estimated Outstanding Access Transactions: 0
Estimated Outstanding Accounting Transactions: 0
Estimated Throttled Access Transactions: 0
Estimated Throttled Accounting Transactions: 0
Maximum Throttled Transactions: access 0, accounting 0
Requests per minute past 24 hours:
high - 0 hours, 40 minutes ago: 5
low - 0 hours, 45 minutes ago: 0
average: 0

Thanks,

Joe

[balaji.bandi]

Can you provide some more information :

show version

show ip route

how is this stack connected in network ? provide that configuraiton of the port (or post running configuration).

is the switch pure Layer 2 ?

10.4.37.151 - is this IP for the ISE ? are you able to ping this from Switch ?

[joeharb]

Cisco IOS XE Software, Version 16.06.06

ValMDF9300#show ip route vrf Mgmt-vrf
Extended Host Mode is enabled

Routing Table: Mgmt-vrf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 10.30.37.1 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.30.37.1

Yes 10.4.37.151 is the ISE server and I can ping it if I specify the VRF.

And to clarify the device uses TACACS that is on the same segment of 10.4.37.X and that is working without issue.  The interfaces are the same on both groups:

aaa group server tacacs+ CSI_BB
server-private 10.4.37.91 key 7 XXXXXXXXXXXXXXXX
server-private 10.30.37.91 key 7 XXXXXXXXXXX
ip vrf forwarding Mgmt-vrf
ip tacacs source-interface GigabitEthernet0/0
!
aaa group server radius ISE
server name PadISE1
ip vrf forwarding Mgmt-vrf
ip radius source-interface GigabitEthernet0/0
load-balance method least-outstanding

Pure Layer 2

Port channel made up 2 Ten Gig ports to 7K that is a VPC.

Thanks,

Joe