cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1328
Views
0
Helpful
3
Replies

WS-C4510R BB1, BB2 High CPU Issue through the Memory Leak

Kiyoun Jang
Cisco Employee
Cisco Employee

Hi Team,

 

i have some question about the issue.

H/W: 4510

S/W: cat4500e-universalk9.SPA.03.03.00.SG.151-1.SG.bin

BB2 was hit the Memory Leak bug and there was log about the SPT resource

 

"Pool: I/O Free: 3584 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Spanning Tree", ipl= 0, pid= 185."

 

Actually there are BB1, BB2 but BB2 got high CPU when type the 'Show Tech' and BB2 go down with high CPU. at the same time BB1 also got high CPU without any of action, but there is no much log.

 

At the during the 'Show tech' in BB2, i can see high number of Sa Miss and STPDrop.

I suspect to Spanning Tree loop at the BB1, BB2 but i cant see any of other log.

so do you think it was the Loop?. i am trying to find out possible way in this issue.

 

SA Miss = This CPU event, is caused when a new MAC is seen on the port, and it is thus, a copy of such packet is forwarded to CPU for MAC learning.

 

TxModeDrop = This copy of packet is then dropped by CPU once it has been processed, and the counter increments. So, when a packet encounters SA miss event, a copy comes to cpu and it is dropped as a TxModeDrop.

 

STPDrop is a drop due to port blocked by topology protocols (like
Spanning Tree, or REP, or FlexLink, etc).

 

<BB2>

------------------ show platform cpu packet statistics ------------------

RkGenericPacketMan:
Packet allocation failures: 0
Packet Buffer(SW Common) allocation failures: 0
Packet Buffer(SW ESMP) allocation failures: 0
Packet Buffer(SW EOBC) allocation failures: 0
Packet Buffer(SW SupToSup) allocation failures: 0

 

Packets Dropped In Processing Overall

Total                5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ----------
             1092858      1209      1044       353         15

Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Sa Miss                        1091173      1209      1036       350         15
L2 Router                         1446         0         0         1          0
Input Acl Fwd                       20         0         0         0          0
Input ACl Copy                     198         0         0         0          0
Sw Packet for Bridge                   21         0         0         0       0

 

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Normal                             120         0         0         0          0
Medium                         1091194      1209      1036       350         15
High                              1544         0         0         1          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
------------------ -------------------- --------- --------- --------- ----------
STPDrop                           49899       412       207        62          1
NoDstPorts                           21         0         0         0          0
Tx Mode Drop                    1042938       796       827       287         12

Total packet queues 64

 

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Input ACL fwd(snooping)        49233216         8         4         3          0
Host Learning                  1093009      1162      1030       351         15
L2 Control                   152411117         6         0         0          0
Input ACL log, unreach         5003068         0         0         0          0
L3 Glean                        412960         0         0         0          0
L3 Receive                    22331201        97        23         2          0
Ttl Expired                          9         0         0         0          0
InputIf Fail                    717145         0         0         0          0
Bfd                             112411         0         0         0          0
L2 router to CPU, 7          904361527       123       938       353         24

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning                      250         0         0         0          0
Input ACL log, unreach           13001         0        68        23          0
L2 router to CPU, 7             285967       808      2189       613          0

 

 

<BB1>

 

Packets Dropped In Processing Overall

 

Total                5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

             1653436         0         0         0          0

 

Packets Dropped In Processing by CPU event

 

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Sa Miss                        1645174         0         0         0          0

Input If Fail                        1         0         0         0          0

L2 Router                         3198         0         0         0          0

L3 Receive                           2         0         0         0          0

Input Acl Fwd                     4943         0         0         0          0

Input ACl Copy                       4         0         0         0          0

Sw Packet for Bridge                  114         0         0         0          0

 

Packets Dropped In Processing by Priority

 

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Normal                            4970         0         0         0          0

Medium                         1645288         0         0         0          0

High                              3178         0         0         0          0

 

Packets Dropped In Processing by Reason

 

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

STPDrop                           23555         0         0         0          0

NoDstPorts                          114         0         0         0          0

Tx Mode Drop                    1629767         0         0         0          0

 

Total packet queues 64

 

Packets Received by Packet Queue

 --More-

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Input ACL fwd(snooping)        63446381         3         1         0          0

Host Learning                  1645146         0         0         0          0

L2 Control                    86166598         2         0         0          0

Input ACL log, unreach         4602642         0         0         0          0

L3 Glean                      24117027         0         0         0          0

L3 Receive                   137395204         6         1         0          0

Ttl Expired                        309         0         0         0          0

InputIf Fail                  67664210         8         0         0          0

Bfd                                  4         0         0         0          0

L2 router to CPU, 7          958344244        35        24        21         15

 

Packets Dropped by Packet Queue

 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Input ACL fwd(snooping)       354724682         0         0         0          0

Host Learning                    14454         0         0         0          0

Input ACL log, unreach           50880         0         0         0          0

InputIf Fail                        22         0         0         0          0

L2 router to CPU, 7           84821357         0         0         0          0

 

Thanks

 

1 Accepted Solution

Accepted Solutions

Matt Delony
Cisco Employee
Cisco Employee

Hello Kiyoun Jang,

 

I/O pool is used for process-switched traffic. If I/O pool running low it can be either memory leak or high rate of traffic to/from CPU.

 

STPdrop is for when traffic gets dropped instead of going out of port in STP blocking state. An example would be broadcast. Broadcast is flooded out of all ports in broadcast domain except ingress port and any STP blocking ports. Broadcast not sent out of STP blocking port is dropped as STPdrop. High rate of STPdrop might indicate large amount of traffic getting flooded.

 

SA miss is for when source MAC address is either not in MAC table or already learned on different port (mac move).

If the issue is still going, you can try turning on mac move notification and check for any mac flaps with switch having high SA miss (mac-address-table notification mac-move).

 

It's hard to say if issue was a switching loop. The evidence points to possible broadcast storm.

View solution in original post

3 Replies 3

Leo Laohoo
Hall of Fame
Hall of Fame

@Kiyoun Jang wrote:

S/W: cat4500e-universalk9.SPA.03.03.00.SG.151-1.SG.bin

 


Whao, IOS-XE version 3.3.0!  

This switch is running a beta IOS.  I'd start thinking about upgrading the firmware.

Matt Delony
Cisco Employee
Cisco Employee

Hello Kiyoun Jang,

 

I/O pool is used for process-switched traffic. If I/O pool running low it can be either memory leak or high rate of traffic to/from CPU.

 

STPdrop is for when traffic gets dropped instead of going out of port in STP blocking state. An example would be broadcast. Broadcast is flooded out of all ports in broadcast domain except ingress port and any STP blocking ports. Broadcast not sent out of STP blocking port is dropped as STPdrop. High rate of STPdrop might indicate large amount of traffic getting flooded.

 

SA miss is for when source MAC address is either not in MAC table or already learned on different port (mac move).

If the issue is still going, you can try turning on mac move notification and check for any mac flaps with switch having high SA miss (mac-address-table notification mac-move).

 

It's hard to say if issue was a switching loop. The evidence points to possible broadcast storm.

thank you!!

Review Cisco Networking for a $25 gift card