08-27-2014 01:03 PM - edited 03-07-2019 08:33 PM
Hello,
we have GRE tunnel over two c6500. But we have a problem with CPU usage (IP input process) at only one border router.
Our conf:
router1:
interface Tunnel1
description test1
ip address xxx.xxx.xxx.226 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source Vlan22
tunnel destination ccc.ccc.ccc.ccc
tunnel path-mtu-discovery
router2(on this router si high CPU utilization:
interface Tunnel1
description test1
ip address xxx.xxx.xxx.225 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source ccc.ccc.ccc.ccc
tunnel destination aaa.aaa.aaa.aaa
tunnel path-mtu-discovery
Log from ip debug:
Log Buffer (8192 bytes):
c=46945, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:09 bts: UDP src=46945, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:09 bts: UDP src=46945, dst=51413
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, sending, proto=47
Aug 27 15:53:09 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
Aug 27 15:53:09 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, MTU Processing(4), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Protocol Output Counter(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Sendself Check(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, HW Shortcut Installation(15), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, sending last fragment
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, post-encap feature, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:09 bts: IP: s=94.254.8.69 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 76, sending last fragment
Aug 27 15:53:09 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 100, sending, proto=47
Aug 27 15:53:09 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:10 bts: UDP src=25766, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:10 bts: UDP src=25766, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:10 bts: UDP src=25766, dst=51413
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, sending, proto=47
Aug 27 15:53:10 bts: FIBipv4-packet-proc: route packet from Null0 src ccc.ccc.ccc.ccc dst aaa.aaa.aaa.aaa
Aug 27 15:53:10 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, MTU Processing(4), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Protocol Output Counter(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, IP Sendself Check(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=ccc.ccc.ccc.ccc (Null0), d=aaa.aaa.aaa.aaa (Vlan1906), len 1420, post-encap feature, proto=47, HW Shortcut Installation(15), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, sending last fragment
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, post-encap feature, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:10 bts: IP: s=178.223.71.226 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 27, sending last fragment
7 15:53:11 bts: UDP src=54646, dst=51413, IP Protocol Output Counter(5), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts: UDP src=54646, dst=51413, IP Sendself Check(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts: UDP src=54646, dst=51413, HW Shortcut Installation(15), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=81.25.184.52 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, sending fragment
Aug 27 15:53:11 bts: UDP src=54646, dst=514137 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb, len 1438, input feature
Aug 27 15:53:11 bts: UDP src=56446, dst=51413, MCI Check(55), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: FIBipv4-packet-proc: route packet from Vlan1906 src 109.228.185.218 dst bbb.bbb.bbb.bbb
Aug 27 15:53:11 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:11 bts: IP: tableid=0, s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), routed via FIB
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts: UDP src=56446, dst=51413, IP Post Routing Processing(22), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts: UDP src=56446, dst=51413, TCP Adjust MSS(38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1438, output feature
Aug 27 15:53:11 bts: UDP src=56446, dst=51413, Post-Ingress-NetFlow(49), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), g=bbb.bbb.bbb.bbb, len 1438, forward
Aug 27 15:53:11 bts: UDP src=56446, dst=51413
Aug 27 15:53:11 bts: IP: s=109.228.185.218 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1396, post-encap feature
Aug 27 15:53:11 bts: UDP src=56446, dst=51413, MTU Processing(4), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
.87 dst bbb.bbb.bbb.bbb
Aug 27 15:53:12 bts: FIBipv4-packet-proc: packet routing succeeded
Aug 27 15:53:12 bts: IP: tableid=0, s=85.226.130.87 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), routed via FIB
Aug 27 15:53:12 bts: IP: s=85.226.130.87 (Vlan1906), d=bbb.bbb.bbb.bbb (Tunnel1), len 1451
08-27-2014 02:28 PM
Hey Josh,
Use the Netdr capture to identify traffic hitting the CPU:
https://supportforums.cisco.com/document/59956/troubleshooting-netdr-capture-sup7206500
HTH.
Regards,
RS.
08-27-2014 04:58 PM
Hello,
thank you for the quick reply.
I used this tool and capture some packets but I don't know how to find problem :-/.
It could be a problem with the MTU and fragmentation?
------- dump of incoming inband packet -------
interface Vl1906, routine mistral_process_rx_packet_inlin, timestamp 01:36:44.719
dbus info: src_vlan 0x772(1906), src_indx 0x82(130), len 0x5BE(1470)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
D0020401 07720400 00820005 BE000000 00110560 0E000040 00000000 03800000
mistral hdr: req_token 0x0(0), src_index 0x82(130), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x772(1906)
destmac 00.YY.YY.YY.YY.YY, srcmac 00.XX.XX.XX.XX.XX, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1452, identifier 59626
df 0, mf 0, fo 0, ttl 54, src aaa.aaa.aaa.aaa, dst bbb.bbb.bbb.bbb
udp src 62558, dst 51413 len 1432 checksum 0x8C37
------- dump of incoming inband packet -------
interface Vl1906, routine mistral_process_rx_packet_inlin, timestamp 01:36:44.719
dbus info: src_vlan 0x772(1906), src_indx 0x82(130), len 0x5B0(1456)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
10020401 07720400 00820005 B0000000 00110550 0E000040 00000000 03800000
mistral hdr: req_token 0x0(0), src_index 0x82(130), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x772(1906)
destmac 00.YY.YY.YY.YY.YY, srcmac 00.XX.XX.XX.XX.XX, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1438, identifier 17960
df 0, mf 0, fo 0, ttl 118, src ccc.ccc.ccc.ccc, dst bbb.bbb.bbb.bbb
udp src 43999, dst 51413 len 1418 checksum 0x3A24
08-27-2014 06:22 PM
Josh,
In the dump I see that traffic is coming from Src index 130. Run the following commands:
#remote command switch test mcast ltl-info index 130
It must list a physical port from where this traffic is coming. Try shutting down that port and let me know if it lowers the CPU.
HTH.
Regards,
RS.
08-27-2014 11:29 PM
Hi,
I'm not sure if it's related to your CPU problems but I wonder why your tunnel interfaces are configured with fixed MTU values and MSS clamping (ip tcp adjust-mss) on the one hand and PMTUD on the other. From what I understand, you should use either one or the other.
With PMTUD, the tunnel interfaces can dynamically adjust their tunnel MTU by copying the original (inner) IP packet's DF-bit into the (outer) tunnel IP header. End systems that also do PMTUD then are able to adjust their MSS to the appropriate values dynamically.
MSS clamping, in contrast, overrides the MSS values exchanged by end systems during the TCP handshakes in order to negotiate the best possible value along the path with the configured (fixed) value. This tricks end system's PMTUD processs.
HTH
Rolf
08-28-2014 07:13 AM
Hello Rolf,
thanks for your replay.
When I use this settings:
router1:
interface Tunnel1
description test1
ip address xxx.xxx.xxx.226 255.255.255.252
tunnel source Vlan22
tunnel destination ccc.ccc.ccc.ccc
tunnel path-mtu-discovery
router2(on this router si high CPU utilization:
interface Tunnel1
description test1
ip address xxx.xxx.xxx.225 255.255.255.252
tunnel source ccc.ccc.ccc.ccc
tunnel destination aaa.aaa.aaa.aaa
tunnel path-mtu-discovery
CPU load is OK, but some pages not working eg. speedtest. net
08-28-2014 07:00 AM
Hello,
you mean
#remote command switch test mcast ltl-info index 82
index 0x82 contain ports 3/3
because
index 0x130 contain ports * empty *
Yes, I understand this, but I can't shutdown this port, because it's my uplink to the Internet.
I think the problem is in the MTU.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide