cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
2114
Views
15
Helpful
7
Replies
BashedRoot
Beginner

High CPU Utilization & Maxed TCAM on 3750G

I would appreciate any help you can offer to optimize the CPU and IOS in general.

I have approximately 10k IPs routed and about 40 VLANS.

Thank you.

Cisco 3750G (WS-C3750G-24T-S)
Cisco IOS 12.2(55)SE5 (IPSERVICESK9)

Cisco3750#show platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                        784/6272        757/5992  
 IPv4 IGMP groups + multicast routes:          152/1216          6/26    
 IPv4 unicast directly-connected routes:       784/6272        757/5992  
 IPv4 unicast indirectly-connected routes:     272/2176         77/556   
 IPv4 policy based routing aces:                 0/0             0/0     
 IPv4 qos aces:                                768/768         260/260   
 IPv4 security aces:                          1024/1024         39/39    

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization
Cisco3750#show sdm prefer routing
 "desktop routing" template:
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  3K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    11K
    number of directly-connected IPv4 hosts:        3K
    number of indirect IPv4 routes:                 8K
  number of IPv4 policy based routing aces:         0.5K
  number of IPv4/MAC qos aces:                      0.5K
  number of IPv4/MAC security aces:                 1K
Cisco3750#show processes cpu sorted 5sec
CPU utilization for five seconds: 71%/30%; one minute: 73%; five minutes: 84%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 297         478        13      36769  6.44%  0.51%  0.10%   3 SSH Process      
  82    33972245   5515061       6159  5.02%  5.74%  8.47%   0 HLFM address lea
 322     8978872   3627822       2475  2.66%  2.53%  1.86%   0 SNMP ENGINE      
 315     6497835    376670      17250  1.57%  1.64%  1.64%   0 CEF: IPv4 proces
  69     5455738     98258      55524  1.57%  1.46%  1.52%   0 Adjust Regions   
 288         486        13      37384  1.25%  0.10%  0.02%   1 SSH Process      
 206     4714523  11732768        401  1.25%  1.26%  1.38%   0 IP Input         
 145     2877795   5340068        538  1.25%  0.83%  0.77%   0 Hulc LED Process
  72      993580   3127121        317  0.62%  0.22%  0.28%   0 hrpc <- response
 106      906572    387005       2342  0.31%  0.20%  0.23%   0 hpm counter proc
 146     3296177   4132574        797  0.31%  0.91%  0.98%   0 HL3U bkgrd proce
 320      667892   2351095        284  0.31%  0.43%  0.24%   0 IP SNMP          
 236       52254    158319        330  0.15%  0.02%  0.00%   0 TCP Protocols    
 155      363878    213594       1703  0.15%  0.09%  0.09%   0 HRPC qos request
  31      123611     35051       3526  0.15%  0.05%  0.01%   0 Net Background   
  71      100660   3162978         31  0.15%  0.01%  0.00%   0 hrpc -> request  
  68      692071  10868364         63  0.15%  0.15%  0.19%   0 Fifo Error Detec
 154      434498     53664       8096  0.15%  0.08%  0.09%   0 HQM Stack Proces
  84      128846   5516082         23  0.15%  0.06%  0.02%   0 HLFM address ret
  20        7369    260640         28  0.00%  0.00%  0.00%   0 IPC Deferred Por
  19           0         6          0  0.00%  0.00%  0.00%   0 IPC Managed Time
  22           0         1          0  0.00%  0.00%  0.00%   0 IPC Session Serv
  21        2452     36029         68  0.00%  0.00%  0.00%   0 IPC Seat Manager
  18        8135    260639         31  0.00%  0.00%  0.00%   0 IPC Periodic Tim
Cisco3750#show controllers cpu-interface 
ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain
-------------------------------------------------------------------------
ASIC0     0          0          0          0          0          0         
ASIC1     0          0          0          0          0          0         
ASIC2     0          0          0          0          0          0         
ASIC3     0          0          0          0          0          0         
ASIC4     0          0          0          0          0          0         
ASIC5     0          0          0          0          0          0         

HOL Fix Counts
--------------
No Fixes:          0 Added:        172 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
   5308098          0          0          0          0          1

Rx Delay
         0          1          2          3          4
---------- ---------- ---------- ---------- ----------
   5308098          0          0          0          0

AddlDelay AdvanceCnt
---------- ----------
         0          0

Rx Retries by RetryCount
         0          1          2          3          4          5          6
---------- ---------- ---------- ---------- ---------- ---------- ----------
   5308098          0          0          0          0          0          0

         7          8          9
---------- ---------- ----------
         0          0          0

AddlRetry
----------
         0


cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc               8590811    0          0          0          0         
stp               4071       0          0          0          0         
ipc               348656     0          0          0          0         
routing protocol  956882     0          0          0          0         
L2 protocol       17178      0          0          0          0         
remote console    0          0          0          0          0         
sw forwarding     513680680  0          0          0          0         
host              18234719   0          0          0          0         
broadcast         96238      0          0          0          0         
cbt-to-spt        0          0          0          0          0         
igmp snooping     593841     0          0          0          0         
icmp              517823747  0          0          172        0         
logging           0          0          0          0          0         
rpf-fail          0          0          0          0          0         
dstats            0          0          0          0          0         
cpu heartbeat     5308098    0          0          0          0         

cpu-queue         static inuse static added
----------------- ------------ ------------
rpc               0            0           
stp               0            0           
ipc               0            0           
routing protocol  0            0           
L2 protocol       0            0           
remote console    0            0           
sw forwarding     0            0           
host              0            0           
broadcast         0            0           
cbt-to-spt        0            0           
igmp snooping     0            0           
icmp              0            172         
logging           0            0           
rpf-fail          0            0           
dstats            0            0           
cpu heartbeat     0            0           

Supervisor ASIC receive-queue parameters
----------------------------------------
 queue 0 maxrecevsize 7E0 pakhead 4749478 paktail 5623C50
 queue 1 maxrecevsize 7E0 pakhead 483582C paktail 57A15E8
 queue 2 maxrecevsize 7E0 pakhead 47E71E8 paktail 47E6E4C
 queue 3 maxrecevsize 7E0 pakhead 58FAD4C paktail 58FA614
 queue 4 maxrecevsize 7E0 pakhead 483A8D8 paktail 57A6C88
 queue 5 maxrecevsize 7E0 pakhead 4ADF734 paktail 58DF300
 queue 6 maxrecevsize 7E0 pakhead 58E88F0 paktail 58E8554
 queue 7 maxrecevsize 7E0 pakhead 5882820 paktail 4A9F450
 queue 8 maxrecevsize 7E0 pakhead 4AA7EBC paktail 4AA6CB0
 queue 9 maxrecevsize 7E0 pakhead 57E9814 paktail 57E9814
 queue A maxrecevsize 7E0 pakhead 498247C paktail 49832EC
 queue B maxrecevsize 7E0 pakhead 5905498 paktail 5904D60
 queue C maxrecevsize 7E0 pakhead 49A3354 paktail 57F99D4
 queue D maxrecevsize 7E0 pakhead 499ACBC paktail 57E9478
 queue E maxrecevsize 0 pakhead 0 paktail 0
 queue F maxrecevsize 7E0 pakhead 57D6008 paktail 497DCD0
Supervisor ASIC exception status
--------------------------------
Receive overrun    00000000   Transmit overrun 00000000
FrameSignatureErr  00000000   MicInitialize    00000000
BadFrameErr        00000000   LenExceededErr   00000000
BadJumboSegments   00000000

Supervisor ASIC Mic Registers
------------------------------
MicDirectPollInfo               80000200
MicIndicationsReceived          00000000
MicInterruptsReceived           00000000
MicPcsInfo                      0000011F
MicPlbMasterConfiguration       00000000
MicRxFifosAvailable             00000040
MicRxFifosReady                 0000BFBF
MicTimeOutPeriod:       FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000
MicTransmFramesCopied           00000003
MicTxFifosAvailable             0000003F
MicConfiguration:       Conf flag: 00000110     Interrupt Flag: 0000000A
MicReceiveFifoAssignmen Queue 0 - 7: 00000000   Queue 8 - 15:00000000
MicReceiveFramesReady:  FrameAvailable: 00000841        frameAvaiMask: 00000000
MicException:    
        Exception_flag  00000000
        Message-1       00000000
        Message-2       00000000
        Message-3       00000000
MicIntRxFifo:
        ReadPtr         000001A8        WritePtr        000001A8
        WHeadPtr        000001A8        TxFifoDepth     C0000800
MicIntTxFifo:
        ReadPtr         00000480        WritePtr        00000480
        WHeadPtr        00000480        TxFifoDepth     C0000800
MicDecodeInfo:
Fifo0:  address:        03FF4000 asic_num:      00000100
Fifo1:  address:        03FF4400 asic_num:      00000101
MicTransmitFifoInfo:
Fifo0:   StartPtrs:     066C9000        ReadPtr:        066C9110
        WritePtrs:      066C9110        Fifo_Flag:      8A800800
        Weights:        001E001E
Fifo1:   StartPtrs:     06427400        ReadPtr:        06427608
        WritePtrs:      06427608        Fifo_Flag:      89800400
        Weights:        000A000A
MicReceiveFifoInfo:
Fifo0:  StartPtr:       066DD000        ReadPtr:        066DD678
        WritePtrs:      066DD6D8        Fifo_Flag:      8B000FA0
        writeHeaderPtr: 066DD6D8
Fifo1:  StartPtr:       069DA800        ReadPtr:        069DAB38
        WritePtrs:      069DAB38        Fifo_Flag:      89800400
        writeHeaderPtr: 069DAB38
Fifo2:  StartPtr:       06993000        ReadPtr:        06993008
        WritePtrs:      06993008        Fifo_Flag:      89800400
        writeHeaderPtr: 06993008
Fifo3:  StartPtr:       06BEF800        ReadPtr:        06BEF800
        WritePtrs:      06BEF800        Fifo_Flag:      89800400
        writeHeaderPtr: 06BEF800
Fifo4:  StartPtr:       06A22000        ReadPtr:        06A220D8
        WritePtrs:      06A220D8        Fifo_Flag:      89800400
        writeHeaderPtr: 06A220D8
Fifo5:  StartPtr:       06B5EE00        ReadPtr:        06B5EE00
        WritePtrs:      06B5EE00        Fifo_Flag:      88800200
        writeHeaderPtr: 06B5EE00
Fifo6:  StartPtr:       06BA5C00        ReadPtr:        06BA5D38
        WritePtrs:      06BA5D38        Fifo_Flag:      89400000
        writeHeaderPtr: 06BA5D38
Fifo7:  StartPtr:       06ACE800        ReadPtr:        06ACE930
        WritePtrs:      06ACE930        Fifo_Flag:      89800400
        writeHeaderPtr: 06ACE930
Fifo8:  StartPtr:       06B38E00        ReadPtr:        06B38F88
        WritePtrs:      06B38F88        Fifo_Flag:      88800200
        writeHeaderPtr: 06B38F88
Fifo9:  StartPtr:       066DB4D8        ReadPtr:        066DB4D8
        WritePtrs:      066DB4D8        Fifo_Flag:      82800008
        writeHeaderPtr: 066DB4D8
Fifo10: StartPtr:       06A6B800        ReadPtr:        06A6B880
        WritePtrs:      06A6B880        Fifo_Flag:      88800200
        writeHeaderPtr: 06A6B880
Fifo11: StartPtr:       066C8F00        ReadPtr:        066C8F20
        WritePtrs:      066C8F60        Fifo_Flag:      86000040
        writeHeaderPtr: 066C8F60
Fifo12: StartPtr:       06A98800        ReadPtr:        06A98B00
        WritePtrs:      06A98800        Fifo_Flag:      89000100
        writeHeaderPtr: 06A98800
Fifo13: StartPtr:       06427880        ReadPtr:        06427880
        WritePtrs:      06427880        Fifo_Flag:      86800080
        writeHeaderPtr: 06427880
Fifo14: StartPtr:       00000000        ReadPtr:        00000000
        WritePtrs:      00000000        Fifo_Flag:      00800000
        writeHeaderPtr: 00000000
Fifo15: StartPtr:       06992F80        ReadPtr:        06992F90
        WritePtrs:      06992F90        Fifo_Flag:      84800020
        writeHeaderPtr: 06992F90


===========================================================
Complete Board Id:0x4003
===========================================================
7 REPLIES 7
Peter Paluch
Hall of Fame Cisco Employee

Hello,

It does appear as if your TCAM utilization was nearing (not hitting yet, though) the available maximums. The CPU load, however, does not appear to be caused by handling traffic in software - the load incurred by the IP Input process seems to be negligible. Are you experiencing any issues besides seeing a relatively high CPU load?

According to what you are saying, your 3750 seems to be in a relatively busy environment, handling many routes and many MAC addresses (can you perhaps post the output of the show ip route summary and show mac address-table count | i Space output)? It might be possible that in this particular application, the 3750 model is strained to its limits, and you may be hitting the maximums of what can be reasonably achieved with this particular switch platform.

Surely, you can change the SDM template to a different setting that would allocate different amounts of TCAM space to different applications. Check the Table 2-23 in the following document for details of individual templates without IPv6 support:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/15-0_2_se/command/reference/cr3750/cli2.html#wp12430152

Unfortunately, the output of show platform tcam utilization you have posted suggests that you are closing to the limits both for MAC addresses and IP route entries. Any SDM template different from desktop default you appear to be running now will only decrease either of these application spaces, and your performance may be severely impacted.

I am afraid that your situation cannot be solved by simple reconfiguration. You are hitting the limits of what your platform can provide. Decreasing either the number of MAC or IP route entries in this switch would give it more breathing room in the TCAM, but doing this may need re-engineering your network (terminating VLANs at a different point, performing route summarization).

My two cents...

Best regards,
Peter

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Peter, in way of background, OP has another post, that got us to the point of this post.

In that post, I too noted TCAM utilization wasn't at the maximums, but I did wonder if perhaps it occasionally did hit/exceed the maximums.  (Not noted in that other reply, but I especially wonder about MACs TCAM utilization - as it might change quite a bit dynamically.)

Although IP Input is low, on a switch like a 3750, I wonder whether even a bit over 1% CPU is truly "negligible" when you consider how much slower, I believe, the 3750 software forwarding is compared to ASIC forwarding.

But perhaps even more important, I also wonder about the 30% CPU utilization shown for interrupt processing, which I also suspect is effectively 3750 software forwarding, even if "fast path".

For comparison, here's stats one of "my" prod 3750-X stacks (of 3):

L3switch#sh proc c s

CPU utilization for five seconds: 28%/0%; one minute: 28%; five minutes: 27%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 159  10763945702071642563        519  5.75%  6.95%  7.03%   0 Hulc LED Process
  75  4047685615 631192272       6412  4.63%  4.62%  4.75%   0 RedEarth Tx Mana
   4  1108601767  49151211      22555  2.87%  1.48%  1.25%   0 Check heaps
  74  17922193051050268881       1706  2.71%  2.54%  2.40%   0 RedEarth I2C dri
 119  1518462554 180111379       8430  1.43%  1.66%  1.70%   0 hpm counter proc
 344   288238626 121857183       2365  0.95%  1.25%  0.66%   0 SNMP ENGINE
 343    30931837 100211186        308  0.63%  0.26%  0.09%   0 PDU DISPATCHER
 342    59898270 206995229        289  0.47%  0.20%  0.10%   0 IP SNMP
 220   174195092 385166282        452  0.47%  0.38%  0.37%   0 Spanning Tree
 115   1647482051429386546        115  0.47%  0.44%  0.47%   0 hpm main process
 170   329024073  17658161      18633  0.31%  0.33%  0.31%   0 HQM Stack Proces
 311     9274835 606336023         15  0.31%  0.06%  0.04%   0 PM Callback
  76    37706818 521984711         72  0.31%  0.05%  0.00%   0 RedEarth Rx Mana
  97    433986752567551623          0  0.31%  0.15%  0.13%   0 HLFM address lea
  96   3207834741519026989        211  0.15%  0.15%  0.16%   0 HRPC hlfm reques
 203     1091608 171104313          6  0.15%  0.01%  0.00%   0 IP ARP Track
 360    252694082482912406          0  0.15%  0.06%  0.05%   0 RADIUS
 272    27613798 195198007        141  0.15%  0.10%  0.10%   0 Marvell wk-a Pow
 165    33434151 875409009         38  0.15%  0.11%  0.12%   0 HSRP Common
  38   181122151  87913419       2060  0.15%  0.26%  0.26%   0 Per-Second Jobs
 305    32799549   6593454       4974  0.15%  0.06%  0.07%   0 Syslog Traps
  70    68352055  17696220       3862  0.15%  0.13%  0.15%   0 Compute load avg
 120    97154350 367686231        264  0.15%  0.11%  0.11%   0 HRPC pm-counters
 201    53403773 290286466        183  0.15%  0.20%  0.11%   0 IP Input
  99    236972552568899285          0  0.15%  0.08%  0.05%   0 HLFM address ret
  25           0         1          0  0.00%  0.00%  0.00%   0 Crash writer

.

.

IP Input seems to bounce zero and under .7%.

Notice my interrupt CPU is 0% and in the above, IP Input only .11% for 5 minutes.

So, Peter, any additional thoughts on OP's interrupt and IP Input CPU utilization?

Peter Paluch
Hall of Fame Cisco Employee

Hi Joe,

Oh, I see. Thank you for providing me with the additional info - I did not know about those things.

In that post, I too noted TCAM utilization wasn't at the maximums, but I did wonder if perhaps it occasionally did hit/exceed the maximums.  (Not noted in that other reply, but I especially wonder about MACs TCAM utilization - as it might change quite a bit dynamically.)

That is a logical thing to assume but there are, in my opinion, two things that speak against these "occassional" spikes as the explanation:

  1. Filling the MAC address region of the TCAM full with MAC addresses is not going to cause the traffic for unknown MAC addresses to be handled by CPU. Instead, the switching hardware will simply flood the frames toward unknown destinations within their respective VLAN. This flooding is handled independently of the CPU. In other words, a frame with an unknown destination address should not be "punted" to the CPU. As far as my memory goes, I haven't seen a CPU load going up just because the switch was forced to flood traffic toward an unknown MAC address..

    I have, however, seen the CPU load going up when experimenting with MAC table overflow attacks. It seems that when the switching hardware receives a frame whose source MAC is unknown, it signals the CPU to program the MAC address at an appropriate TCAM address. I do not suppose that the CPU is handling the entire frame; rather, I would believe that the switching hardware keeps the unknown MAC address in a standalone control register or an internal buffer where the CPU can find it during the event handling. So an inrush of lots of frames with unknown source MACs can cause the CPU go up because of the need for the CPU to program these MACs into the TCAM; however, the CPU should not be handling the flooding of the frames themselves.

    In fact, the issue with MAC address learning shall be investigated closer - the PID 82, "HLFM address lea", is the process performing MAC address learning, and it occupies 5%-8% of the CPU which is quite a lot for a stable network.

  2. If the CPU was handling IP traffic which indeed gets punted to the CPU for various reasons including TCAM space shortage, either in software CEF (interrupts) or process switching (IP Input), then the exhaustion of the TCAM by IP route entries would definitely be logged. The message would probably be "PLATFORM_UCAST-6-PREFIX: One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix" or a different message suggesting that a prefix didn't make it into the TCAM. The original poster does not mention whether such message has appeared in the logs. However, "spikes" in the number of IP routes possibly exhausting the TCAM space would not go unnoticed.

It may be interesting why there is enough IP traffic to warrant the "IP Input" and "CEF: IPv4 proces" processes to consume over 1% each, but even then, it's just 2% of the process CPU load.

There could be something going on - the 30% of interrupt load must stand for something, and I honestly do not know what they could be. My recommendation would be to look both into the possibility of MAC address flooding attacks or perhaps Port Security kicking into action often (if configured), and into the the possibility of whether the IP traffic is being CPU-switches, and if so, why. It would be useful to see the outputs of the show cef not-cef-switches and show ip cef switching statistics feature commands, perhaps also show ip traffic as well.

Best regards,
Peter

Thanks for your help everyone, really appreciate it!

Cisco3750#show cef not-cef-switch  
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag
RP         0       0      518363        0        7   350073        0        0
2          0       0           0        0        0        0        0        0
Cisco3750#show ip cef switching statistics feature
IPv4 CEF input features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
Total                            0          0          0          0          0

IPv4 CEF output features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF post-encap features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF for us features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF punt features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF local features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
Total                            0          0          0          0          0
Cisco3750#show ip traffic
IP statistics:
  Rcvd:  57319099 total, 55083372 local destination
         0 format errors, 0 checksum errors, 1883599 bad hop count
         3 unknown protocol, 76 not a gateway
         0 security failures, 0 bad options, 356100 with options
  Opts:  355638 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 355612 record route
         0 stream ID, 0 strict source route, 462 alert, 0 cipso, 0 ump
         78 other
  Frags: 52 reassembled, 1108 timeouts, 0 couldn't reassemble
         0 fragmented, 0 couldn't fragment
  Bcast: 46506500 received, 0 sent
  Mcast: 0 received, 0 sent
  Sent:  22103584 generated, 2076720241 forwarded
  Drop:  0 encapsulation failed, 0 unresolved, 2624 no adjacency
         4 no route, 0 unicast RPF, 0 forced drop
         0 options denied, 3 source IP address zero

ICMP statistics:
  Rcvd: 111146 format errors, 324 checksum errors, 370 redirects, 49458 unreachable
        1596288 echo, 1508 echo reply, 0 mask requests, 0 mask replies, 20 quench
        0 parameter, 10 timestamp, 0 info request, 2 other
        0 irdp solicitations, 1 irdp advertisements
  Sent: 0 redirects, 159340 unreachable, 0 echo, 1596288 echo reply
        0 mask requests, 0 mask replies, 0 quench, 10 timestamp
        0 info reply, 13793255 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 4129875 total, 1115 checksum errors, 87711 no port
  Sent: 2993126 total

UDP statistics:
  Rcvd: 49189703 total, 371 checksum errors, 45597274 no port
  Sent: 3561571 total, 0 forwarded broadcasts

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

EIGRP-IPv4 statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0

OSPF statistics:
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
  Rcvd: 331665 requests, 531260 replies, 0 reverse, 0 other
  Sent: 8100118 requests, 287986 replies (206585 proxy), 0 reverse
  Drop due to input queue full: 34788

#1 Peter, couldn't the switch maintain a larger RAM based MAC address table, to deal with TCAM overflow?  If so, MAC lookup outside the TCAM should consume more CPU.  From a utilization basis, shouldn't be nearly as intense as route lookup, but I would presume a lot slower than a TCAM lookup.

Peter Paluch
Hall of Fame Cisco Employee

Hi Joe,

#1 Peter, couldn't the switch maintain a larger RAM based MAC address table, to deal with TCAM overflow?

It could - but to my best knowledge, it does not do it. You see, the lack of a MAC address entry in TCAM is not nearly as critical as the lack of an IP prefix entry in TCAM. If a particular destination MAC address is missing, the switch will resort to flooding the frame, and this should handled by the switching hardware, not CPU. The recipient will still get its frame. Apart from the flooding which could be perceived as a drawback, there are no ill effects from missing a particular destination MAC address in TCAM - most importantly, it does not create reachability issues so there's no point in inventing fallback scenarios.

IP prefixes are a different story - if a particular destination IP prefix is not programmed into TCAM then the switch cannot route packets toward that destination network. That is, I believe, the reason why multilayer switches go to such lengths as installing wildcard punt entries into TCAM if the space is being exhausted to allow the packets to be handled by software - so that the destination still remains reachable.

Best regards,
Peter

Peter, I fully agree, MAC TCAM failure not likely to be as much of an issue as route TCAM failure (BTW, we've had 3750s do the latter, and performance tanks).

Also agreed, we don't know if a 3750 maintains a RAM based MAC table too, but I suspect you may be right, as there's not as much need as there is with routes.  If that's the case, I also agree with you, it probably make little difference to CPU consumption.

However, "Apart from the flooding which could be perceived as a drawback, . . ", perhaps it might be so perceived, laugh.

Anyway, for the OP, Peter's initial suggestion, that you may need to do some network re-engineering, has much merit.  Or, as we also discussed, you might need to use different hardware.  Understand the 3750 series wasn't really designed for demanding L2 or L3, except for 3750G-12S which was designed for a distribution role.  (Which is why it has additional TCAM resources and additional SDM templates.)

The newer 3650 and 3850 might also be someone lacking in their support for a large number of MACs and/or routes.  The 4500 and 4900 series were designed for more demanding usage (as are the switches designed for MetroE).