7609 High CPU : punted packets to RP CPU

[Dung Nguyen Anh]

Dear all,

We have a serious problem related to CPU 7609 router that affects on bussiness. CPU utilization is always approximate ~ 90%.

we do troubleshooting with Debug netdr capture and show netdr capture we see many dump messages with different IP src and IP dst as following:

------- dump of incoming inband packet -------
interface NULL, routine mistral_process_rx_packet_inlin, timestamp 17:09:20.208
dbus info: src_vlan 0x403(1027), src_indx 0x1C3(451), len 0x52(82)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  92020C01 04030400 01C30000 52000000 00060448 28000040 00000000 03800000 
mistral hdr: req_token 0x0(0), src_index 0x1C3(451), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x403(1027)
destmac 00.1C.0F.5E.E0.80, srcmac 00.00.00.00.00.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x48, totlen 64, identifier 54542
  df 1, mf 0, fo 0, ttl 64, src 10.164.224.158, dst 173.194.72.101
    tcp src 56964, dst 443, seq 2669801125, ack 0, win 65535 off 11 checksum 0xB005 syn

------- dump of outgoing inband packet -------
interface Te8/1, routine draco2_fastsend, timestamp 17:09:20.208
dbus info: src_vlan 0x42E(1070), src_indx 0x1C3(451), len 0x52(82)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
  02020000 042E2800 01C30000 52000000 00060448 20000000 00000000 03800000 
mistral hdr: req_token 0x0(0), src_index 0x1C3(451), rx_offset 0x76(118)
  requeue 0, obl_pkt 0, vlan 0x403(1027)
destmac 2C.21.72.AE.58.53, srcmac 00.1C.0F.5E.E0.80, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x48, totlen 64, identifier 54542
  df 1, mf 0, fo 0, ttl 63, src 10.164.224.158, dst 173.194.72.101
    tcp src 56964, dst 443, seq 2669801125, ack 0, win 65535 off 11 checksum 0xB027 syn

I don't understand where have a srcmac 00.00.00.00.00.00, and how does it affect on my CPU router?

Please suggest me how to solve this issue? 

Tks,

Dzung

[smith.billy]

Dear All,

We have the same issue with our 7606. CPU usage jumps to 60-70% each evening.

Let me give two most peculiar examples from TX netdr capture:

1. Packet coming from strange vlan 1027 (same as in Dzung post), not even present in the vlan database on the router:

interface Gi5/2, routine process_rx_packet_inline, timestamp 08:26:23.876
dbus info: src_vlan 0x403(1027), src_indx 0x101(257), len 0x40(64)
 bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
 50000401 04030000 01010000 40000000 0011D59C 78D973EF E7402800 03800000
destmac 00.1E.F7.F6.2B.40, srcmac 00.24.C4.C0.AA.40, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 36, identifier 8404
 df 0, mf 0, fo 0, ttl 243, src 115.239.231.64, dst 213.150.120.217
   udp src 80, dst 123 len 16 checksum 0x0

2. Src MAC all zeros, unknown protocol  006A hex

------- dump of incoming inband packet -------
interface Vl2, routine process_rx_packet_inline, timestamp 09:48:34.204
dbus info: src_vlan 0x2(2), src_indx 0x387(903), len 0x7C(124)
  bpdu 0, index_dir 0, flood 0, dont_lrn 1, dest_indx 0x380(896)
  F0020808 0002A800 03870000 7C000000 9E000000 20000000 00000000 03800000
destmac 00.1E.F7.F6.2B.40, srcmac 00.00.00.00.00.00, protocol 006A
layer 3 data: AAAA0300 000C0113 02000900 20016C00 000012E1 06D59C76
              BCBC284F C20050EA 3A100000 00000000 00000000 00000000
              00000000 00000000 000002E1 00000344 00000380 7ACA

Can someone give some insight on the issue ?

Thanks in advance,

Marek.

gambuzo at gmail dot com

[Nikhil Kapoor]

Hello Marek, 

interface Gi5/2, routine process_rx_packet_inline, timestamp 08:26:23.876
dbus info: src_vlan 0x403(1027), src_indx 0x101(257), len 0x40(64)
 bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
 50000401 04030000 01010000 40000000 0011D59C 78D973EF E7402800 03800000
destmac 00.1E.F7.F6.2B.40, srcmac 00.24.C4.C0.AA.40, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 36, identifier 8404
 df 0, mf 0, fo 0, ttl 243, src 115.239.231.64, dst 213.150.120.217
   udp src 80, dst 123 len 16 checksum 0x0

Based on the above netdr capture I could draw following observations. 

1. Its an IP packet. 
2. Based on the destination Index (0x380). That confirms that its unicast packet is being punted to the RP. 
3. Source and destination IP address 115.239.231.64 & 213.150.120.217. 
4. Port incolved udp src 80 and dst 123 (NTP). 

To isolate the things. Please check and see if the router has a valid path for below IP address to transit through it. 

Also if these IPs resides on router? 

The packets are received from Vlan 1027. Please check why we are receiving such packets from Vlan 1027 which are being punted to RP though the TTL is 243 which should be transit. 

show ip ro 115.239.231.64
show ip ro 213.150.120.217

If possible try to shut vlan 1027 and check if things gets resolved. 

HTH, 
Nikhil