Nexus 3064 CoPP copp-s-glean high Drops causing CPU spike

satish.txt1 · ‎05-28-2018

This is very interesting thing i found, i am seeing glean counter dropping millions of packet very ~60 second around and during that time very high CPU spiked noticed on proc cpu history graph.

# show policy-map interface control-plane
...
...
class-map copp-s-glean (match-any)
      police pps 500
        OutPackets    286253
        DropPackets   1243234301
...
...

After googling found its related to arp flood in network and could be spanning tree related, This is what i am seeing on switches related spanning tree events.

# show spanning-tree internal event-history all

206) Transition at 196700 usecs after Mon May 28 11:57:32 2018
     Root: 80ca.2c86.d283.4680 Cost: 2 Age:  1 Root Port: Ethernet1/10 Port: Ethernet1/36 [STP_TREE_EV_MULTI_FLUSH_RCVD]

207) Transition at 224153 usecs after Mon May 28 11:57:33 2018
     Root: 80ca.2c86.d283.4680 Cost: 2 Age:  1 Root Port: Ethernet1/10 Port: Ethernet1/36 [STP_TREE_EV_MULTI_FLUSH_RCVD]

208) Transition at 199688 usecs after Mon May 28 11:58:49 2018
     Root: 80ca.2c86.d283.4680 Cost: 2 Age:  1 Root Port: Ethernet1/10 Port: Ethernet1/36 [STP_TREE_EV_MULTI_FLUSH_RCVD]

209) Transition at 199563 usecs after Mon May 28 11:58:51 2018
     Root: 80ca.2c86.d283.4680 Cost: 2 Age:  1 Root Port: Ethernet1/10 Port: Ethernet1/36 [STP_TREE_EV_MULTI_FLUSH_RCVD]

210) Transition at 204056 usecs after Mon May 28 12:00:08 2018
     Root: 80ca.2c86.d283.4680 Cost: 2 Age:  1 Root Port: Ethernet1/10 Port: Ethernet1/36 [STP_TREE_EV_MULTI_FLUSH_RCVD]

We have bunch of Cisco 3850 cat switches also in network and i don't know what command i should use on them to get spanning tree history related spanning tree issue ( I am less familiar to cat OS )

Or any other best way to find out source of origin to find arp flood.

Georg Pauwen · ‎05-29-2018

Hello,

on the 3850, use the command 'show spanning-tree detail'.

That said, if possible, post a drawing of the logical setup of your network. You want to make sure that the root switches are correct...

satish.txt1 · ‎07-04-2018

I found issue, one of HOST connected to switch port somewhere in switch fabric was making STP topology change, server was rebooting every 1 minute and creating spanning tree change notification because of that switch was flushing ARP table. after shutdown bad host server port everything was claim down.