3850 stack with very high packet punt numbers

[tzpatrick]

I'm running 16.12.05b, and the switch is constantly maxing out CPU resources with "IOSXE-RP Punt Se" in the upper 50th percentiles.

Digging into it (by running "show platform software fed switch active punt cause summary"), I see a lot of packets with Cause 58 ("Layer2 bridge domain data packe") even after clearing it. There's also a good amount of Cause 96 ("Layer2 control protocols"), but only around 1% of Cause 58.

The larger setup is Meraki MX firewalls running in HA through multiple 3850 switch stacks used only for layer 2 (running ipbasek9 since they were layer 3 in the past). This is the only stack with the high CPU utilization problem.

I've reloaded it hoping it would at least clear up any queues or memory, but upon booting back up, the same issues come up.

Does anyone have advice on the next steps to try to troubleshot this or what could be causing this?

[MHM Cisco World]

can I see 
show ip traffic statistic 
MHM

[tzpatrick]

show ip traffic doesn't have "statistics" as an option, but here is the "show ip traffic" result:

IP statistics:
Rcvd: 14825 total, 1947 local destination
0 format errors, 0 checksum errors, 130 bad hop count
0 unknown protocol, 4 not a gateway
0 security failures, 0 bad options, 36576 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 36576 alert, 0 cipso, 0 ump
0 other, 0 ignored
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
0 invalid hole
Bcast: 18 received, 0 sent
Mcast: 14386 received, 27 sent
Sent: 2639 generated, 0 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 2563 no adjacency
0 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr
0 options denied, 0 source IP address zero

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
287 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
0 time exceeded, 0 info replies
Sent: 0 redirects, 0 unreachable, 0 echo, 89 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 166 total, 0 checksum errors, 2 no port 0 finput
Sent: 189 total, 0 forwarded broadcasts

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
Report VRF mismatch drops: 0

TCP statistics:
Rcvd: 1494 total, 0 checksum errors, 4 no port
Sent: 2361 total

EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total

ARP statistics:
Rcvd: 6911 requests, 20 replies, 0 reverse, 0 other
Sent: 6 requests, 973 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

---

[MHM Cisco World]

clear ip traffic 
the share show again let see which count increase rapidly 
MHM

[tzpatrick]

done. Within a minute and a half, this is what it looked like:

IP statistics:
Rcvd: 184 total, 169 local destination
0 format errors, 0 checksum errors, 2 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 595 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 595 alert, 0 cipso, 0 ump
0 other, 0 ignored
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
0 invalid hole
Bcast: 0 received, 0 sent
Mcast: 182 received, 0 sent
Sent: 126 generated, 0 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 146 no adjacency
0 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr
0 options denied, 0 source IP address zero

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
0 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
0 time exceeded, 0 info replies
Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 2 total, 0 checksum errors, 0 no port 0 finput
Sent: 2 total, 0 forwarded broadcasts

OSPF statistics:
Last clearing of OSPF traffic counters 00:01:30
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
Report VRF mismatch drops: 0

TCP statistics:
Rcvd: 167 total, 0 checksum errors, 0 no port
Sent: 124 total

EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total

ARP statistics:
Rcvd: 81 requests, 0 replies, 0 reverse, 0 other
Sent: 0 requests, 9 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

---

[MHM Cisco World]

there are many count 
let start with ARP are you see incomplete in show arp ?
MHM

[MHM Cisco World]

Local destiantion and option and alert count is increasing.

These all related to packet with options and packet with options like mpls maybe show these traffic.

MHM

[tzpatrick]

---

@MHM Cisco World wrote:

there are many count 
let start with ARP are you see incomplete in show arp ?
MHM

---

The arp only shows the IPs in the management VLAN for the switch:

1. The gateway of the VLAN (managed and configured on the MX)
2. The IP for the other switch stack connected to it
3. IPs of the 4 WAPs

The routing itself is actually handled by the Meraki MX connected via a trunk

[MHM Cisco World]

show ip traffic <interface connect to Meraki MX>
check if the Meraki is source of these option packet
MHM