11-12-2019 10:44 PM
Hi Team,
i have some question about the issue.
H/W: 4510
S/W: cat4500e-universalk9.SPA.03.03.00.SG.151-1.SG.bin
BB2 was hit the Memory Leak bug and there was log about the SPT resource
"Pool: I/O Free: 3584 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Spanning Tree", ipl= 0, pid= 185."
Actually there are BB1, BB2 but BB2 got high CPU when type the 'Show Tech' and BB2 go down with high CPU. at the same time BB1 also got high CPU without any of action, but there is no much log.
At the during the 'Show tech' in BB2, i can see high number of Sa Miss and STPDrop.
I suspect to Spanning Tree loop at the BB1, BB2 but i cant see any of other log.
so do you think it was the Loop?. i am trying to find out possible way in this issue.
SA Miss = This CPU event, is caused when a new MAC is seen on the port, and it is thus, a copy of such packet is forwarded to CPU for MAC learning.
TxModeDrop = This copy of packet is then dropped by CPU once it has been processed, and the counter increments. So, when a packet encounters SA miss event, a copy comes to cpu and it is dropped as a TxModeDrop.
STPDrop is a drop due to port blocked by topology protocols (like
Spanning Tree, or REP, or FlexLink, etc).
<BB2>
------------------ show platform cpu packet statistics ------------------
RkGenericPacketMan:
Packet allocation failures: 0
Packet Buffer(SW Common) allocation failures: 0
Packet Buffer(SW ESMP) allocation failures: 0
Packet Buffer(SW EOBC) allocation failures: 0
Packet Buffer(SW SupToSup) allocation failures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
1092858 1209 1044 353 15
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss 1091173 1209 1036 350 15
L2 Router 1446 0 0 1 0
Input Acl Fwd 20 0 0 0 0
Input ACl Copy 198 0 0 0 0
Sw Packet for Bridge 21 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Normal 120 0 0 0 0
Medium 1091194 1209 1036 350 15
High 1544 0 0 1 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
STPDrop 49899 412 207 62 1
NoDstPorts 21 0 0 0 0
Tx Mode Drop 1042938 796 827 287 12
Total packet queues 64
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping) 49233216 8 4 3 0
Host Learning 1093009 1162 1030 351 15
L2 Control 152411117 6 0 0 0
Input ACL log, unreach 5003068 0 0 0 0
L3 Glean 412960 0 0 0 0
L3 Receive 22331201 97 23 2 0
Ttl Expired 9 0 0 0 0
InputIf Fail 717145 0 0 0 0
Bfd 112411 0 0 0 0
L2 router to CPU, 7 904361527 123 938 353 24
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning 250 0 0 0 0
Input ACL log, unreach 13001 0 68 23 0
L2 router to CPU, 7 285967 808 2189 613 0
<BB1>
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
1653436 0 0 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss 1645174 0 0 0 0
Input If Fail 1 0 0 0 0
L2 Router 3198 0 0 0 0
L3 Receive 2 0 0 0 0
Input Acl Fwd 4943 0 0 0 0
Input ACl Copy 4 0 0 0 0
Sw Packet for Bridge 114 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Normal 4970 0 0 0 0
Medium 1645288 0 0 0 0
High 3178 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
STPDrop 23555 0 0 0 0
NoDstPorts 114 0 0 0 0
Tx Mode Drop 1629767 0 0 0 0
Total packet queues 64
Packets Received by Packet Queue
--More-
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping) 63446381 3 1 0 0
Host Learning 1645146 0 0 0 0
L2 Control 86166598 2 0 0 0
Input ACL log, unreach 4602642 0 0 0 0
L3 Glean 24117027 0 0 0 0
L3 Receive 137395204 6 1 0 0
Ttl Expired 309 0 0 0 0
InputIf Fail 67664210 8 0 0 0
Bfd 4 0 0 0 0
L2 router to CPU, 7 958344244 35 24 21 15
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping) 354724682 0 0 0 0
Host Learning 14454 0 0 0 0
Input ACL log, unreach 50880 0 0 0 0
InputIf Fail 22 0 0 0 0
L2 router to CPU, 7 84821357 0 0 0 0
Thanks
Solved! Go to Solution.
11-13-2019 02:28 PM - edited 11-13-2019 02:34 PM
Hello Kiyoun Jang,
I/O pool is used for process-switched traffic. If I/O pool running low it can be either memory leak or high rate of traffic to/from CPU.
STPdrop is for when traffic gets dropped instead of going out of port in STP blocking state. An example would be broadcast. Broadcast is flooded out of all ports in broadcast domain except ingress port and any STP blocking ports. Broadcast not sent out of STP blocking port is dropped as STPdrop. High rate of STPdrop might indicate large amount of traffic getting flooded.
SA miss is for when source MAC address is either not in MAC table or already learned on different port (mac move).
If the issue is still going, you can try turning on mac move notification and check for any mac flaps with switch having high SA miss (mac-address-table notification mac-move).
It's hard to say if issue was a switching loop. The evidence points to possible broadcast storm.
11-13-2019 01:04 AM
@Kiyoun Jang wrote:
S/W: cat4500e-universalk9.SPA.03.03.00.SG.151-1.SG.bin
Whao, IOS-XE version 3.3.0!
This switch is running a beta IOS. I'd start thinking about upgrading the firmware.
11-13-2019 02:28 PM - edited 11-13-2019 02:34 PM
Hello Kiyoun Jang,
I/O pool is used for process-switched traffic. If I/O pool running low it can be either memory leak or high rate of traffic to/from CPU.
STPdrop is for when traffic gets dropped instead of going out of port in STP blocking state. An example would be broadcast. Broadcast is flooded out of all ports in broadcast domain except ingress port and any STP blocking ports. Broadcast not sent out of STP blocking port is dropped as STPdrop. High rate of STPdrop might indicate large amount of traffic getting flooded.
SA miss is for when source MAC address is either not in MAC table or already learned on different port (mac move).
If the issue is still going, you can try turning on mac move notification and check for any mac flaps with switch having high SA miss (mac-address-table notification mac-move).
It's hard to say if issue was a switching loop. The evidence points to possible broadcast storm.
11-14-2019 12:51 AM
thank you!!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide