7609-S high CPU utilization due to IP Input

[oscar.garciaj8]

Hi!

I have a Cisco 7609-S and recently experimented high CPU utilization. I attach an important information:

Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Thu 15-Mar-12 08:07 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE6, RELEASE SOFTWARE (fc1)

RM-MTY-01 uptime is 3 years, 29 weeks, 3 days, 7 hours, 46 minutes
Uptime for this control processor is 3 years, 29 weeks, 3 days, 4 hours, 27 minutes
System returned to ROM by power cycle (SP by power on)
System restarted at 08:12:35 CST Wed Jan 30 2019
System image file is "disk0:c7600s72033-advipservicesk9-mz.122-33.SRE6.bin"
Last reload type: Normal Reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco CISCO7609-S (R7000) processor (revision 1.0) with 983008K/65536K bytes of memory.
Processor board ID FOX11110L6L
SR71000 CPU at 600MHz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
22 Virtual Ethernet interfaces
148 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

------------------ show process cpu history ------------------

999999999999999999999999999999999999999999999999999999999999
866666888888888889999999998888999998888888888888886666668888
100 **********************************************************
90 **********************************************************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

999999999999999999999999999999999999999999999999999999999999
999999999999999987999999999999889998999999999899899999999999
100 *#*##**#***************#********##******##****#***#*******
90 ##########################################################
80 ##########################################################
70 ##########################################################
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

999999999999999999999999999999999999999999999999999999999999999999999999
999999999999999999999999999999999999999999999999999999999999999999999999
100 **********************************************************************
90 #######***********##########******************************************
80 ###############################******************#********************
70 ################################****************####*#*#*#************
60 ######################################################################
50 ######################################################################
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

RM-MTY-01#show processes cpu sorted
CPU utilization for five seconds: 62%/47%; one minute: 78%; five minutes: 80%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
220 1973557112 3640113288 0 5.27% 5.40% 5.42% 0 IP Input
267 95668 126861 754 0.71% 2.63% 2.81% 1 Virtual Exec
12 644210716 793653526 811 0.71% 0.68% 0.65% 0 ARP Input
268 185841784 536975016 346 0.55% 0.51% 0.50% 0 Ethernet OAM Pro
213 923390972 995653054 927 0.55% 0.69% 0.67% 0 ADJ resolve proc
564 570645604 1705060643 334 0.47% 0.46% 0.41% 0 PDU DISPATCHER
548 205415980 969531015 211 0.47% 0.46% 0.47% 0 NHRP
557 26138296 2191722850 0 0.31% 0.30% 0.31% 0 HSRP Common

Please help me to resolve this issue.

Thanks in advance.

[MHM Cisco World]

but friend there are many interface with Jumbo MTU frame size forward to SP via MTU 1500?

[oscar.garciaj8]

So for example in this interface  GigabitEthernet1/26 we must decrease MTU value from 9000 to 1500?

interface GigabitEthernet1/26
description BB:RMMTY1_A_RAMTY3
mtu 9000
ip address 189.206.235.57 255.255.255.252
no ip redirects
ip ospf authentication message-digest
ip ospf message-digest-key 100 md5 <removed>
ip ospf network point-to-point
ip ospf cost 75
logging event link-status
load-interval 30
speed 1000
duplex full
mpls ip
mpls label protocol ldp

As i understood it remains in this way:

interface GigabitEthernet1/26
description BB:RMMTY1_A_RAMTY3
mtu 1500
ip address 189.206.235.57 255.255.255.252
no ip redirects
ip ospf authentication message-digest
ip ospf message-digest-key 100 md5 <removed>
ip ospf network point-to-point
ip ospf cost 75
logging event link-status
load-interval 30
speed 1000
duplex full
mpls ip
mpls label protocol ldp

Thanks in advance.

[MHM Cisco World]

I can not recommend that, 
there is absolutely reason the MTU is config with this value,
you must discus this setting with your team. 

[oscar.garciaj8]

So, what is your recommendation then?

I do not understand.

[MHM Cisco World]

first confirm that you use interface with MTU 1500 as tunnel source of GRE tunnel 
let explain 
you have two server farm in two VLAN and inter-VLAN done in router, both interface for VLAN is support Jumbo 9000 mtu size
so change this MTU size to make router not fragment packet when forward traffic via tunnel is OK BUT it reduce the speed inside you network, 

what I meaning
traffic VLAN->VLAN since both interface support 9000 size then keep it and get high speed INSIDE your network 
traffic VLAN->GRE tunnel this traffic since the GRE tunnel is 1500 then MTU mismatch force router to fragment the packet. 

so how we save Speed and in same time keep high Speed inside our network, I think a lot, 
then come idea 
9000 size frame come from which protocol 
70-80% come from TCP 
so instead change the MTU of VLAN we only force Router to adjust TCP in GRE interface (tunnel)
this make router receive max 1500 (send from Server) and no need anymore to fragment. 
I do lab and test adjust of tcp mss 

Hope you read my Post carefully and read it again and if you see that is right to apply to your case try it. 

[oscar.garciaj8]

Thanks for your comments. And hoy do I force router to adjunto TCP in GRE tunnel?

Thanks in advance.

[MHM Cisco World]

before apply the command make sure the MTU is 1500 for tunnel and tunnel source 
ip tcp adjust-mss 1400 

[oscar.garciaj8]

OK, it config apply in all the interfaces GRE tunnel?

I must write the config after do that ? Or do I need to do something else?

Thanks in advance.

[MHM Cisco World]

NO, need to write, 
it must take effect immediately.
do you check CPU ?

[oscar.garciaj8]

Ok, so just to confirm, it config must be set on the interfaces that have Jumbo 9000 MTU size?

Also, the value of 1400 must be set the same on all the GRE interfaces ?

Do you think we'll present some affectation on the service affter or during applying the configuration?

I must be crear all my doubts before we proceed to set this configuration.

Thanks in advance.

Thanks in advance.

[MHM Cisco World]

I will send you photo with detail. 
Yes check and check and check before apply any command. 
You are totally right. 

[MHM Cisco World]

Two case the light blue line show your traffic, 
Case1
if the traffic between your Core/Dis SW and Router and then from router to SP (via GRE tunnel ) to other side, and your SP support only 1500, the Edge router will fragment the traffic and High CPU utilize, so we need to make Edge Router adjust the TCP MSS to inform both client and Server of TCP session that the max MTU I can handle is 1400. 

Case2 
the traffic never out Your Edge router, i.e. not pass to SP, so you can free config the MTU to be 9000 if the Core/Dis SW and your Edge router is support jumbo (and Here I ask you to discus with your team why jumbo frame inside your network)
here we dont need to adjust the MSS. 

By the way why jumbo frame all 9216 except two interface 9000 ?

[oscar.garciaj8]

I'm going to discuss this with my team and let you know if the configuration was set. In case we decide to not apply this config, you think is a good idea to reload the router?

Thanks in advance.

[oscar.garciaj8]

Hi

Thanks for your comments. What would be the risk or cons to set this configuration on the interfaces?

Shall I apply this in a maintenance window?

Thanks in advance.

[kirankelkar10]

brilliant @MHM Cisco World