Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1070
Views
0
Helpful
6
Replies

Cisco 6500 SUP720-3B GRE High CPU Utilization in IP Input Process

JoshIIDIF
Level 1
Level 1

‎08-27-2014 01:03 PM - edited ‎03-07-2019 08:33 PM

Hello,

we have GRE tunnel over two c6500.  But we have a problem with CPU usage (IP input process) at only one border router.

Our conf:

router1:

interface Tunnel1
 description test1
 ip address xxx.xxx.xxx.226 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source Vlan22
 tunnel destination ccc.ccc.ccc.ccc
 tunnel path-mtu-discovery

router2(on this router si high CPU utilization:

interface Tunnel1
 description test1
 ip address xxx.xxx.xxx.225 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source ccc.ccc.ccc.ccc
 tunnel destination aaa.aaa.aaa.aaa
 tunnel path-mtu-discovery

 

Log from ip debug:

Log Buffer (8192 bytes):
c=46945, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:09 bts:     UDP src=46945, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:09 bts:     UDP src=46945, dst=51413
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, sending, proto=47
Aug 27 15:53:09 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
Aug 27 15:53:09 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, MTU Processing(4), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Protocol Output Counter(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Sendself Check(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, HW Shortcut Installation(15), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, sending last fragment
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, sending last fragment
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 100, sending, proto=47
Aug 27 15:53:09 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
FALSE     
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:10 bts:     UDP src=25766, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:10 bts:     UDP src=25766, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:10 bts:     UDP src=25766, dst=51413
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, sending, proto=47
Aug 27 15:53:10 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
Aug 27 15:53:10 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, MTU Processing(4), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Protocol Output Counter(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Sendself Check(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, HW Shortcut Installation(15), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, sending last fragment
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, sending last fragment
7 15:53:11 bts:     UDP src=54646, dst=51413, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts:     UDP src=54646, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts:     UDP src=54646, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:11 bts:     UDP src=54646, dst=514137 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb, len 1438, input feature
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413, MCI Check(55), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: FIBipv4-packet-proc: route packet from Vlan1906 src 109.228.185.218 dst bbb.bbb.bbb.bbb
Aug 27 15:53:11 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:11 bts: IP: tableid=0, s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), routed via FIB
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413, IP Post Routing Processing(22), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413, TCP Adjust MSS(38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413, Post-Ingress-NetFlow(49), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), g=bbb.bbb.bbb.bbb, len 1438, forward
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts:     UDP src=56446, dst=51413, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
.87 dst bbb.bbb.bbb.bbb
Aug 27 15:53:12 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:12 bts: IP: tableid=0, s=85.226.130.87 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), routed via FIB
Aug 27 15:53:12 bts: IP: s=85.226.130.87 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1451

 

 

Labels:
0 Helpful
6 Replies 6

Rajeev Sharma
Cisco Employee
Cisco Employee

‎08-27-2014 02:28 PM

Hey Josh,

Use the Netdr capture to identify traffic hitting the CPU:

https://supportforums.cisco.com/document/59956/troubleshooting-netdr-capture-sup7206500

HTH.

Regards,

RS.

0 Helpful

JoshIIDIF
Level 1
Level 1
In response to Rajeev Sharma

‎08-27-2014 04:58 PM

Hello,

thank you for the quick reply.

I used this tool and capture some packets but I don't know how to find problem :-/.

It could be a problem with the MTU and fragmentation?

 

------- dump of incoming inband packet -------
interface Vl1906, routine mistral_process_rx_packet_inlin, timestamp 01:36:44.719
dbus info: src_vlan 0x772(1906), src_indx 0x82(130), len 0x5BE(1470)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  D0020401 07720400 00820005 BE000000 00110560 0E000040 00000000 03800000 
mistral hdr: req_token 0x0(0), src_index 0x82(130), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x772(1906)
destmac 00.YY.YY.YY.YY.YY, srcmac 00.XX.XX.XX.XX.XX, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1452, identifier 59626
  df 0, mf 0, fo 0, ttl 54, src aaa.aaa.aaa.aaa, dst bbb.bbb.bbb.bbb
    udp src 62558, dst 51413 len 1432 checksum 0x8C37

------- dump of incoming inband packet -------
interface Vl1906, routine mistral_process_rx_packet_inlin, timestamp 01:36:44.719
dbus info: src_vlan 0x772(1906), src_indx 0x82(130), len 0x5B0(1456)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  10020401 07720400 00820005 B0000000 00110550 0E000040 00000000 03800000 
mistral hdr: req_token 0x0(0), src_index 0x82(130), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x772(1906)
destmac 00.YY.YY.YY.YY.YY, srcmac 00.XX.XX.XX.XX.XX, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1438, identifier 17960
  df 0, mf 0, fo 0, ttl 118, src ccc.ccc.ccc.ccc, dst bbb.bbb.bbb.bbb
    udp src 43999, dst 51413 len 1418 checksum 0x3A24

0 Helpful

Rajeev Sharma
Cisco Employee
Cisco Employee
In response to JoshIIDIF

‎08-27-2014 06:22 PM

Josh,

In the dump I see that traffic is coming from Src index 130. Run the following commands:

#remote command switch test mcast ltl-info index 130

It must list a physical port from where this traffic is coming. Try shutting down that port and let me know if it lowers the CPU.

HTH.

Regards,

RS.

0 Helpful

Rolf Fischer
Level 9
Level 9
In response to Rajeev Sharma

‎08-27-2014 11:29 PM

Hi,

I'm not sure if it's related to your CPU problems but I wonder why your tunnel interfaces are configured with fixed MTU values and MSS clamping (ip tcp adjust-mss) on the one hand and PMTUD on the other. From what I understand, you should use either one or the other.

With PMTUD, the tunnel interfaces can dynamically adjust their tunnel MTU by copying the original (inner) IP packet's DF-bit into the (outer) tunnel IP header. End systems that also do PMTUD then are able to adjust their MSS to the appropriate values dynamically.

MSS clamping, in contrast, overrides the MSS values exchanged by end systems during the TCP handshakes in order to negotiate the best possible value along the path with the configured (fixed) value. This tricks end system's PMTUD processs.

 

HTH

Rolf

0 Helpful

JoshIIDIF
Level 1
Level 1
In response to Rolf Fischer

‎08-28-2014 07:13 AM

Hello Rolf,

thanks for your replay.

When I use this settings:

router1:

interface Tunnel1
 description test1
 ip address xxx.xxx.xxx.226 255.255.255.252
 tunnel source Vlan22
 tunnel destination ccc.ccc.ccc.ccc
 tunnel path-mtu-discovery

router2(on this router si high CPU utilization:

interface Tunnel1
 description test1
 ip address xxx.xxx.xxx.225 255.255.255.252
 tunnel source ccc.ccc.ccc.ccc
 tunnel destination aaa.aaa.aaa.aaa
 tunnel path-mtu-discovery

CPU load is OK, but some pages not working eg. speedtest. net

0 Helpful

JoshIIDIF
Level 1
Level 1
In response to Rajeev Sharma

‎08-28-2014 07:00 AM

Hello,

you mean

#remote command switch test mcast ltl-info index 82 

index 0x82 contain ports 3/3

because 

index 0x130 contain ports * empty *

Yes, I understand this, but I can't shutdown this port, because it's my uplink to the Internet.

I think the problem is in the MTU.

 

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card