- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU Utilization & Maxed TCAM on 3750G
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-30-2015 07:57 AM - edited 03-05-2019 03:01 AM
I would appreciate any help you can offer to optimize the CPU and IOS in general.
I have approximately 10k IPs routed and about 40 VLANS.
Thank you.
Cisco 3750G (WS-C3750G-24T-S)
Cisco IOS 12.2(55)SE5 (IPSERVICESK9)
Cisco3750#show platform tcam utilization
CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values
Unicast mac addresses: 784/6272 757/5992
IPv4 IGMP groups + multicast routes: 152/1216 6/26
IPv4 unicast directly-connected routes: 784/6272 757/5992
IPv4 unicast indirectly-connected routes: 272/2176 77/556
IPv4 policy based routing aces: 0/0 0/0
IPv4 qos aces: 768/768 260/260
IPv4 security aces: 1024/1024 39/39
Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization
Cisco3750#show sdm prefer routing
"desktop routing" template:
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.
number of unicast mac addresses: 3K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 11K
number of directly-connected IPv4 hosts: 3K
number of indirect IPv4 routes: 8K
number of IPv4 policy based routing aces: 0.5K
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 1K
Cisco3750#show processes cpu sorted 5sec
CPU utilization for five seconds: 71%/30%; one minute: 73%; five minutes: 84%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
297 478 13 36769 6.44% 0.51% 0.10% 3 SSH Process
82 33972245 5515061 6159 5.02% 5.74% 8.47% 0 HLFM address lea
322 8978872 3627822 2475 2.66% 2.53% 1.86% 0 SNMP ENGINE
315 6497835 376670 17250 1.57% 1.64% 1.64% 0 CEF: IPv4 proces
69 5455738 98258 55524 1.57% 1.46% 1.52% 0 Adjust Regions
288 486 13 37384 1.25% 0.10% 0.02% 1 SSH Process
206 4714523 11732768 401 1.25% 1.26% 1.38% 0 IP Input
145 2877795 5340068 538 1.25% 0.83% 0.77% 0 Hulc LED Process
72 993580 3127121 317 0.62% 0.22% 0.28% 0 hrpc <- response
106 906572 387005 2342 0.31% 0.20% 0.23% 0 hpm counter proc
146 3296177 4132574 797 0.31% 0.91% 0.98% 0 HL3U bkgrd proce
320 667892 2351095 284 0.31% 0.43% 0.24% 0 IP SNMP
236 52254 158319 330 0.15% 0.02% 0.00% 0 TCP Protocols
155 363878 213594 1703 0.15% 0.09% 0.09% 0 HRPC qos request
31 123611 35051 3526 0.15% 0.05% 0.01% 0 Net Background
71 100660 3162978 31 0.15% 0.01% 0.00% 0 hrpc -> request
68 692071 10868364 63 0.15% 0.15% 0.19% 0 Fifo Error Detec
154 434498 53664 8096 0.15% 0.08% 0.09% 0 HQM Stack Proces
84 128846 5516082 23 0.15% 0.06% 0.02% 0 HLFM address ret
20 7369 260640 28 0.00% 0.00% 0.00% 0 IPC Deferred Por
19 0 6 0 0.00% 0.00% 0.00% 0 IPC Managed Time
22 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv
21 2452 36029 68 0.00% 0.00% 0.00% 0 IPC Seat Manager
18 8135 260639 31 0.00% 0.00% 0.00% 0 IPC Periodic Tim
Cisco3750#show controllers cpu-interface
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
ASIC3 0 0 0 0 0 0
ASIC4 0 0 0 0 0 0
ASIC5 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 172 In Use: 0 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
5308098 0 0 0 0 1
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
5308098 0 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5 6
---------- ---------- ---------- ---------- ---------- ---------- ----------
5308098 0 0 0 0 0 0
7 8 9
---------- ---------- ----------
0 0 0
AddlRetry
----------
0
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 8590811 0 0 0 0
stp 4071 0 0 0 0
ipc 348656 0 0 0 0
routing protocol 956882 0 0 0 0
L2 protocol 17178 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 513680680 0 0 0 0
host 18234719 0 0 0 0
broadcast 96238 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 593841 0 0 0 0
icmp 517823747 0 0 172 0
logging 0 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 5308098 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 172
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
Supervisor ASIC receive-queue parameters
----------------------------------------
queue 0 maxrecevsize 7E0 pakhead 4749478 paktail 5623C50
queue 1 maxrecevsize 7E0 pakhead 483582C paktail 57A15E8
queue 2 maxrecevsize 7E0 pakhead 47E71E8 paktail 47E6E4C
queue 3 maxrecevsize 7E0 pakhead 58FAD4C paktail 58FA614
queue 4 maxrecevsize 7E0 pakhead 483A8D8 paktail 57A6C88
queue 5 maxrecevsize 7E0 pakhead 4ADF734 paktail 58DF300
queue 6 maxrecevsize 7E0 pakhead 58E88F0 paktail 58E8554
queue 7 maxrecevsize 7E0 pakhead 5882820 paktail 4A9F450
queue 8 maxrecevsize 7E0 pakhead 4AA7EBC paktail 4AA6CB0
queue 9 maxrecevsize 7E0 pakhead 57E9814 paktail 57E9814
queue A maxrecevsize 7E0 pakhead 498247C paktail 49832EC
queue B maxrecevsize 7E0 pakhead 5905498 paktail 5904D60
queue C maxrecevsize 7E0 pakhead 49A3354 paktail 57F99D4
queue D maxrecevsize 7E0 pakhead 499ACBC paktail 57E9478
queue E maxrecevsize 0 pakhead 0 paktail 0
queue F maxrecevsize 7E0 pakhead 57D6008 paktail 497DCD0
Supervisor ASIC exception status
--------------------------------
Receive overrun 00000000 Transmit overrun 00000000
FrameSignatureErr 00000000 MicInitialize 00000000
BadFrameErr 00000000 LenExceededErr 00000000
BadJumboSegments 00000000
Supervisor ASIC Mic Registers
------------------------------
MicDirectPollInfo 80000200
MicIndicationsReceived 00000000
MicInterruptsReceived 00000000
MicPcsInfo 0000011F
MicPlbMasterConfiguration 00000000
MicRxFifosAvailable 00000040
MicRxFifosReady 0000BFBF
MicTimeOutPeriod: FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000
MicTransmFramesCopied 00000003
MicTxFifosAvailable 0000003F
MicConfiguration: Conf flag: 00000110 Interrupt Flag: 0000000A
MicReceiveFifoAssignmen Queue 0 - 7: 00000000 Queue 8 - 15:00000000
MicReceiveFramesReady: FrameAvailable: 00000841 frameAvaiMask: 00000000
MicException:
Exception_flag 00000000
Message-1 00000000
Message-2 00000000
Message-3 00000000
MicIntRxFifo:
ReadPtr 000001A8 WritePtr 000001A8
WHeadPtr 000001A8 TxFifoDepth C0000800
MicIntTxFifo:
ReadPtr 00000480 WritePtr 00000480
WHeadPtr 00000480 TxFifoDepth C0000800
MicDecodeInfo:
Fifo0: address: 03FF4000 asic_num: 00000100
Fifo1: address: 03FF4400 asic_num: 00000101
MicTransmitFifoInfo:
Fifo0: StartPtrs: 066C9000 ReadPtr: 066C9110
WritePtrs: 066C9110 Fifo_Flag: 8A800800
Weights: 001E001E
Fifo1: StartPtrs: 06427400 ReadPtr: 06427608
WritePtrs: 06427608 Fifo_Flag: 89800400
Weights: 000A000A
MicReceiveFifoInfo:
Fifo0: StartPtr: 066DD000 ReadPtr: 066DD678
WritePtrs: 066DD6D8 Fifo_Flag: 8B000FA0
writeHeaderPtr: 066DD6D8
Fifo1: StartPtr: 069DA800 ReadPtr: 069DAB38
WritePtrs: 069DAB38 Fifo_Flag: 89800400
writeHeaderPtr: 069DAB38
Fifo2: StartPtr: 06993000 ReadPtr: 06993008
WritePtrs: 06993008 Fifo_Flag: 89800400
writeHeaderPtr: 06993008
Fifo3: StartPtr: 06BEF800 ReadPtr: 06BEF800
WritePtrs: 06BEF800 Fifo_Flag: 89800400
writeHeaderPtr: 06BEF800
Fifo4: StartPtr: 06A22000 ReadPtr: 06A220D8
WritePtrs: 06A220D8 Fifo_Flag: 89800400
writeHeaderPtr: 06A220D8
Fifo5: StartPtr: 06B5EE00 ReadPtr: 06B5EE00
WritePtrs: 06B5EE00 Fifo_Flag: 88800200
writeHeaderPtr: 06B5EE00
Fifo6: StartPtr: 06BA5C00 ReadPtr: 06BA5D38
WritePtrs: 06BA5D38 Fifo_Flag: 89400000
writeHeaderPtr: 06BA5D38
Fifo7: StartPtr: 06ACE800 ReadPtr: 06ACE930
WritePtrs: 06ACE930 Fifo_Flag: 89800400
writeHeaderPtr: 06ACE930
Fifo8: StartPtr: 06B38E00 ReadPtr: 06B38F88
WritePtrs: 06B38F88 Fifo_Flag: 88800200
writeHeaderPtr: 06B38F88
Fifo9: StartPtr: 066DB4D8 ReadPtr: 066DB4D8
WritePtrs: 066DB4D8 Fifo_Flag: 82800008
writeHeaderPtr: 066DB4D8
Fifo10: StartPtr: 06A6B800 ReadPtr: 06A6B880
WritePtrs: 06A6B880 Fifo_Flag: 88800200
writeHeaderPtr: 06A6B880
Fifo11: StartPtr: 066C8F00 ReadPtr: 066C8F20
WritePtrs: 066C8F60 Fifo_Flag: 86000040
writeHeaderPtr: 066C8F60
Fifo12: StartPtr: 06A98800 ReadPtr: 06A98B00
WritePtrs: 06A98800 Fifo_Flag: 89000100
writeHeaderPtr: 06A98800
Fifo13: StartPtr: 06427880 ReadPtr: 06427880
WritePtrs: 06427880 Fifo_Flag: 86800080
writeHeaderPtr: 06427880
Fifo14: StartPtr: 00000000 ReadPtr: 00000000
WritePtrs: 00000000 Fifo_Flag: 00800000
writeHeaderPtr: 00000000
Fifo15: StartPtr: 06992F80 ReadPtr: 06992F90
WritePtrs: 06992F90 Fifo_Flag: 84800020
writeHeaderPtr: 06992F90
===========================================================
Complete Board Id:0x4003
===========================================================
- Labels:
-
Other Routing
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2016 06:12 AM
Hello,
It does appear as if your TCAM utilization was nearing (not hitting yet, though) the available maximums. The CPU load, however, does not appear to be caused by handling traffic in software - the load incurred by the IP Input process seems to be negligible. Are you experiencing any issues besides seeing a relatively high CPU load?
According to what you are saying, your 3750 seems to be in a relatively busy environment, handling many routes and many MAC addresses (can you perhaps post the output of the show ip route summary and show mac address-table count | i Space output)? It might be possible that in this particular application, the 3750 model is strained to its limits, and you may be hitting the maximums of what can be reasonably achieved with this particular switch platform.
Surely, you can change the SDM template to a different setting that would allocate different amounts of TCAM space to different applications. Check the Table 2-23 in the following document for details of individual templates without IPv6 support:
Unfortunately, the output of show platform tcam utilization you have posted suggests that you are closing to the limits both for MAC addresses and IP route entries. Any SDM template different from desktop default you appear to be running now will only decrease either of these application spaces, and your performance may be severely impacted.
I am afraid that your situation cannot be solved by simple reconfiguration. You are hitting the limits of what your platform can provide. Decreasing either the number of MAC or IP route entries in this switch would give it more breathing room in the TCAM, but doing this may need re-engineering your network (terminating VLANs at a different point, performing route summarization).
My two cents...
Best regards,
Peter
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2016 07:06 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
Peter, in way of background, OP has another post, that got us to the point of this post.
In that post, I too noted TCAM utilization wasn't at the maximums, but I did wonder if perhaps it occasionally did hit/exceed the maximums. (Not noted in that other reply, but I especially wonder about MACs TCAM utilization - as it might change quite a bit dynamically.)
Although IP Input is low, on a switch like a 3750, I wonder whether even a bit over 1% CPU is truly "negligible" when you consider how much slower, I believe, the 3750 software forwarding is compared to ASIC forwarding.
But perhaps even more important, I also wonder about the 30% CPU utilization shown for interrupt processing, which I also suspect is effectively 3750 software forwarding, even if "fast path".
For comparison, here's stats one of "my" prod 3750-X stacks (of 3):
L3switch#sh proc c s
CPU utilization for five seconds: 28%/0%; one minute: 28%; five minutes: 27%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
159 10763945702071642563 519 5.75% 6.95% 7.03% 0 Hulc LED Process
75 4047685615 631192272 6412 4.63% 4.62% 4.75% 0 RedEarth Tx Mana
4 1108601767 49151211 22555 2.87% 1.48% 1.25% 0 Check heaps
74 17922193051050268881 1706 2.71% 2.54% 2.40% 0 RedEarth I2C dri
119 1518462554 180111379 8430 1.43% 1.66% 1.70% 0 hpm counter proc
344 288238626 121857183 2365 0.95% 1.25% 0.66% 0 SNMP ENGINE
343 30931837 100211186 308 0.63% 0.26% 0.09% 0 PDU DISPATCHER
342 59898270 206995229 289 0.47% 0.20% 0.10% 0 IP SNMP
220 174195092 385166282 452 0.47% 0.38% 0.37% 0 Spanning Tree
115 1647482051429386546 115 0.47% 0.44% 0.47% 0 hpm main process
170 329024073 17658161 18633 0.31% 0.33% 0.31% 0 HQM Stack Proces
311 9274835 606336023 15 0.31% 0.06% 0.04% 0 PM Callback
76 37706818 521984711 72 0.31% 0.05% 0.00% 0 RedEarth Rx Mana
97 433986752567551623 0 0.31% 0.15% 0.13% 0 HLFM address lea
96 3207834741519026989 211 0.15% 0.15% 0.16% 0 HRPC hlfm reques
203 1091608 171104313 6 0.15% 0.01% 0.00% 0 IP ARP Track
360 252694082482912406 0 0.15% 0.06% 0.05% 0 RADIUS
272 27613798 195198007 141 0.15% 0.10% 0.10% 0 Marvell wk-a Pow
165 33434151 875409009 38 0.15% 0.11% 0.12% 0 HSRP Common
38 181122151 87913419 2060 0.15% 0.26% 0.26% 0 Per-Second Jobs
305 32799549 6593454 4974 0.15% 0.06% 0.07% 0 Syslog Traps
70 68352055 17696220 3862 0.15% 0.13% 0.15% 0 Compute load avg
120 97154350 367686231 264 0.15% 0.11% 0.11% 0 HRPC pm-counters
201 53403773 290286466 183 0.15% 0.20% 0.11% 0 IP Input
99 236972552568899285 0 0.15% 0.08% 0.05% 0 HLFM address ret
25 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
.
.
IP Input seems to bounce zero and under .7%.
Notice my interrupt CPU is 0% and in the above, IP Input only .11% for 5 minutes.
So, Peter, any additional thoughts on OP's interrupt and IP Input CPU utilization?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2016 01:32 PM
Hi Joe,
Oh, I see. Thank you for providing me with the additional info - I did not know about those things.
In that post, I too noted TCAM utilization wasn't at the maximums, but I did wonder if perhaps it occasionally did hit/exceed the maximums. (Not noted in that other reply, but I especially wonder about MACs TCAM utilization - as it might change quite a bit dynamically.)
That is a logical thing to assume but there are, in my opinion, two things that speak against these "occassional" spikes as the explanation:
- Filling the MAC address region of the TCAM full with MAC addresses is not going to cause the traffic for unknown MAC addresses to be handled by CPU. Instead, the switching hardware will simply flood the frames toward unknown destinations within their respective VLAN. This flooding is handled independently of the CPU. In other words, a frame with an unknown destination address should not be "punted" to the CPU. As far as my memory goes, I haven't seen a CPU load going up just because the switch was forced to flood traffic toward an unknown MAC address..
I have, however, seen the CPU load going up when experimenting with MAC table overflow attacks. It seems that when the switching hardware receives a frame whose source MAC is unknown, it signals the CPU to program the MAC address at an appropriate TCAM address. I do not suppose that the CPU is handling the entire frame; rather, I would believe that the switching hardware keeps the unknown MAC address in a standalone control register or an internal buffer where the CPU can find it during the event handling. So an inrush of lots of frames with unknown source MACs can cause the CPU go up because of the need for the CPU to program these MACs into the TCAM; however, the CPU should not be handling the flooding of the frames themselves.
In fact, the issue with MAC address learning shall be investigated closer - the PID 82, "HLFM address lea", is the process performing MAC address learning, and it occupies 5%-8% of the CPU which is quite a lot for a stable network. - If the CPU was handling IP traffic which indeed gets punted to the CPU for various reasons including TCAM space shortage, either in software CEF (interrupts) or process switching (IP Input), then the exhaustion of the TCAM by IP route entries would definitely be logged. The message would probably be "PLATFORM_UCAST-6-PREFIX: One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix" or a different message suggesting that a prefix didn't make it into the TCAM. The original poster does not mention whether such message has appeared in the logs. However, "spikes" in the number of IP routes possibly exhausting the TCAM space would not go unnoticed.
It may be interesting why there is enough IP traffic to warrant the "IP Input" and "CEF: IPv4 proces" processes to consume over 1% each, but even then, it's just 2% of the process CPU load.
There could be something going on - the 30% of interrupt load must stand for something, and I honestly do not know what they could be. My recommendation would be to look both into the possibility of MAC address flooding attacks or perhaps Port Security kicking into action often (if configured), and into the the possibility of whether the IP traffic is being CPU-switches, and if so, why. It would be useful to see the outputs of the show cef not-cef-switches and show ip cef switching statistics feature commands, perhaps also show ip traffic as well.
Best regards,
Peter
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2016 01:43 PM
Thanks for your help everyone, really appreciate it!
Cisco3750#show cef not-cef-switch
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'
IPv4 CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 0 0 518363 0 7 350073 0 0
2 0 0 0 0 0 0 0 0
Cisco3750#show ip cef switching statistics feature
IPv4 CEF input features:
Feature Drop Consume Punt Punt2Host Gave route
Total 0 0 0 0 0
IPv4 CEF output features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0
IPv4 CEF post-encap features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0
IPv4 CEF for us features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0
IPv4 CEF punt features:
Feature Drop Consume Punt Punt2Host New i/f
Total 0 0 0 0 0
IPv4 CEF local features:
Feature Drop Consume Punt Punt2Host Gave route
Total 0 0 0 0 0
Cisco3750#show ip traffic
IP statistics:
Rcvd: 57319099 total, 55083372 local destination
0 format errors, 0 checksum errors, 1883599 bad hop count
3 unknown protocol, 76 not a gateway
0 security failures, 0 bad options, 356100 with options
Opts: 355638 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 355612 record route
0 stream ID, 0 strict source route, 462 alert, 0 cipso, 0 ump
78 other
Frags: 52 reassembled, 1108 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 46506500 received, 0 sent
Mcast: 0 received, 0 sent
Sent: 22103584 generated, 2076720241 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 2624 no adjacency
4 no route, 0 unicast RPF, 0 forced drop
0 options denied, 3 source IP address zero
ICMP statistics:
Rcvd: 111146 format errors, 324 checksum errors, 370 redirects, 49458 unreachable
1596288 echo, 1508 echo reply, 0 mask requests, 0 mask replies, 20 quench
0 parameter, 10 timestamp, 0 info request, 2 other
0 irdp solicitations, 1 irdp advertisements
Sent: 0 redirects, 159340 unreachable, 0 echo, 1596288 echo reply
0 mask requests, 0 mask replies, 0 quench, 10 timestamp
0 info reply, 13793255 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 4129875 total, 1115 checksum errors, 87711 no port
Sent: 2993126 total
UDP statistics:
Rcvd: 49189703 total, 371 checksum errors, 45597274 no port
Sent: 3561571 total, 0 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
OSPF statistics:
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
ARP statistics:
Rcvd: 331665 requests, 531260 replies, 0 reverse, 0 other
Sent: 8100118 requests, 287986 replies (206585 proxy), 0 reverse
Drop due to input queue full: 34788
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2016 05:22 PM
#1 Peter, couldn't the switch maintain a larger RAM based MAC address table, to deal with TCAM overflow? If so, MAC lookup outside the TCAM should consume more CPU. From a utilization basis, shouldn't be nearly as intense as route lookup, but I would presume a lot slower than a TCAM lookup.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-05-2016 01:06 AM
Hi Joe,
#1 Peter, couldn't the switch maintain a larger RAM based MAC address table, to deal with TCAM overflow?
It could - but to my best knowledge, it does not do it. You see, the lack of a MAC address entry in TCAM is not nearly as critical as the lack of an IP prefix entry in TCAM. If a particular destination MAC address is missing, the switch will resort to flooding the frame, and this should handled by the switching hardware, not CPU. The recipient will still get its frame. Apart from the flooding which could be perceived as a drawback, there are no ill effects from missing a particular destination MAC address in TCAM - most importantly, it does not create reachability issues so there's no point in inventing fallback scenarios.
IP prefixes are a different story - if a particular destination IP prefix is not programmed into TCAM then the switch cannot route packets toward that destination network. That is, I believe, the reason why multilayer switches go to such lengths as installing wildcard punt entries into TCAM if the space is being exhausted to allow the packets to be handled by software - so that the destination still remains reachable.
Best regards,
Peter
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-05-2016 05:57 AM
Peter, I fully agree, MAC TCAM failure not likely to be as much of an issue as route TCAM failure (BTW, we've had 3750s do the latter, and performance tanks).
Also agreed, we don't know if a 3750 maintains a RAM based MAC table too, but I suspect you may be right, as there's not as much need as there is with routes. If that's the case, I also agree with you, it probably make little difference to CPU consumption.
However, "Apart from the flooding which could be perceived as a drawback, . . ", perhaps it might be so perceived, laugh.
Anyway, for the OP, Peter's initial suggestion, that you may need to do some network re-engineering, has much merit. Or, as we also discussed, you might need to use different hardware. Understand the 3750 series wasn't really designed for demanding L2 or L3, except for 3750G-12S which was designed for a distribution role. (Which is why it has additional TCAM resources and additional SDM templates.)
The newer 3650 and 3850 might also be someone lacking in their support for a large number of MACs and/or routes. The 4500 and 4900 series were designed for more demanding usage (as are the switches designed for MetroE).
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
