HIGH CPU ISSUE IN ROUTER

tac · ‎07-16-2013

Hi Guys

I need urgent help as on emy 7600 router is facing high CPU utilizations, we did many troubleshooting but still no improvement. Please help its live environment.

tac · ‎07-16-2013

Guys Please if reuired I cant post logs required, Suspecting some undue traffic issue, but not able to pinpoint the issue. Please help

tac · ‎07-16-2013

Output of Debug ntdr Capture, If Iam removing the route pointing towards gi6/9.34, processes are going completly normal. Need to identify what packets are causing this High CPU. Please help.

ME-AGGREGATION-RTR#show netdr captured-packets

A total of 4096 packets have been captured

The capture buffer wrapped 0 times

Total capture capacity: 4096 packets

------- dump of outgoing inband packet -------

interface Gi2/0/4, routine cwan_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x470(1136), src_indx 0x380(896), len 0x4D(77)

bpdu 0, index_dir 1, flood 0, dont_lrn 1, dest_indx 0x7F(127)

06028018 04707800 03800000 4D000000 00000000 00000000 00000000 007F0000

mistral hdr: req_token 0x0(0), src_index 0x380(896), rx_offset 0x30(48)

requeue 0, obl_pkt 0, vlan 0x0(0)

destmac 00.24.51.6C.DC.1B, srcmac 00.21.D8.CB.56.00, protocol 0800

layer 3 data: 45C0003B 11CC4000 FF069084 47056415 4705E68C 0017D4D8

13027615 8B2D0B48 50180FA6 2F5A0000 4D452D41 47475245

47415449 4F4E2D52 00000470 00000141 00004518 0808

------- dump of outgoing inband packet -------

interface Gi2/0/4, routine draco2_ibc_soutput, timestamp 16:48:23.220

dbus info: src_vlan 0x470(1136), src_indx 0x380(896), len 0x4D(77)

bpdu 0, index_dir 1, flood 0, dont_lrn 1, dest_indx 0x7F(127)

06028018 04707800 03800000 4D000000 00000000 00000000 00000000 007F0000

mistral hdr: req_token 0x0(0), src_index 0x380(896), rx_offset 0x30(48)

requeue 0, obl_pkt 0, vlan 0x0(0)

destmac 00.24.51.6C.DC.1B, srcmac 00.21.D8.CB.56.00, protocol 0800

layer 3 data: 45C0003B 11CC4000 FF069084 47056415 4705E68C 0017D4D8

13027615 8B2D0B48 50180FA6 2F5A0000 4D452D41 47475245

47415449 4F4E2D52 00000470 00000141 00004518 0808

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

28020800 044F0000 00FF0000 40000000 00060478 20000400 00000000 03805DF0

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 182.161.88.111, dst 118.69.172.247

tcp src 3300, dst 80, seq 131787, ack 0, win 16384 off 5 checksum 0x2D94 syn

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060478 20000000 00000000 03805DF0

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 182.161.88.111, dst 118.69.172.247

tcp src 3300, dst 80, seq 131787, ack 0, win 16384 off 5 checksum 0x2D94 syn

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

38020800 044F0000 00FF0000 40000000 00060478 20000400 00000000 03802DF0

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 182.161.88.111, dst 118.69.172.247

tcp src 3300, dst 80, seq 131787, ack 0, win 16384 off 5 checksum 0x2D94 syn

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060478 20000000 00000000 03802DF0

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 182.161.88.111, dst 118.69.172.247

tcp src 3300, dst 80, seq 131787, ack 0, win 16384 off 5 checksum 0x2D94 syn

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

88020800 044F0000 00FF0000 40000000 00060458 20000400 00000000 038095F6

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 203.8.172.201, dst 118.69.172.122

tcp src 17576, dst 80, seq 114148, ack 0, win 16384 off 5 checksum 0xD272 sy

n

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060458 20000000 00000000 038095F6

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 203.8.172.201, dst 118.69.172.122

tcp src 17576, dst 80, seq 114148, ack 0, win 16384 off 5 checksum 0xD272 sy

n

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

C0020800 044F0000 00FF0000 40000000 00060458 20000400 00000000 0380EDF3

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 203.8.172.201, dst 118.69.172.122

tcp src 17576, dst 80, seq 114148, ack 0, win 16384 off 5 checksum 0xD272 sy

n

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060458 20000000 00000000 0380EDF3

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 203.8.172.201, dst 118.69.172.122

tcp src 17576, dst 80, seq 114148, ack 0, win 16384 off 5 checksum 0xD272 sy

n

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

C8020800 044F0000 00FF0000 40000000 00060418 20000400 00000000 038005EE

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 103.21.148.238, dst 118.69.172.122

tcp src 32553, dst 80, seq 112454, ack 0, win 16384 off 5 checksum 0x1A5E sy

n

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060418 20000000 00000000 038005EE

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 103.21.148.238, dst 118.69.172.122

tcp src 32553, dst 80, seq 112454, ack 0, win 16384 off 5 checksum 0x1A5E sy

n

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

D0020800 044F0000 00FF0000 40000000 00060478 20000400 00000000 038035FF

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 103.10.90.211, dst 118.69.172.122

tcp src 2899, dst 80, seq 112432, ack 0, win 16384 off 5 checksum 0xC870 syn

------- dump of outgoing inband packet -------

interface Gi6/9.34, routine draco2_fastsend, timestamp 16:48:23.220

dbus info: src_vlan 0x22(34), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

00020000 00222800 00FF0000 40000000 00060478 20000000 00000000 038035FF

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.09.0F.88.36.87, srcmac 00.21.D8.CB.56.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 125, src 103.10.90.211, dst 118.69.172.122

tcp src 2899, dst 80, seq 112432, ack 0, win 16384 off 5 checksum 0xC870 syn

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin, timestamp 16:48:23.220

dbus info: src_vlan 0x44F(1103), src_indx 0xFF(255), len 0x40(64)

bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

D8020800 044F0000 00FF0000 40000000 00060478 20000400 00000000 03800DFF

mistral hdr: req_token 0x0(0), src_index 0xFF(255), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0x44F(1103)

destmac 00.21.D8.CB.56.00, srcmac 00.00.00.00.00.00, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 40, identifier 256

df 1, mf 0, fo 0, ttl 126, src 103.10.90.211, dst 118.69.172.122

tcp src 2899, dst 80, seq 112432, ack 0, win 16384 off 5 checksum 0xC870 syn

brett.harding · ‎07-16-2013

As a start, can you attach more output?

show tech

show proc cpu sorted

show interface

show log

jgroselle · ‎08-01-2013

Sounds like you have DOS attack inside your network, probably a computer with a nasty virus. You need to start tracing your network backwards to see what port the attacks are coming from.

You need to look for the port with excessive packets use the command: show int | in is up | rate :on the next switch back to look for ports that have a high amount of packet traffic. If the port with excessive packets is a switch that is daisy chained off the port with the high packet traffic keep repeating until you can isolate the problem to a single device. Unplug the device and see if you CPU usage drops.