Solved: RP drops / Input drops on 6500 Sup 720 10GE

torstefan · ‎06-18-2012

Dear Community,

What does RP drops entail?

We are experiencing 0.23 % drops on an vlan int on one of our 6500. This interface is an L3 to a server-subnet.

29563976÷12851487771×100 = 0.230043218

Doing a sh int vlan 238 switching shows 30935411 RP (Route Processing) drops.

It was a bit unclear to me in the thread "Input queue drops on 6500 vlan interface" if RP drops was through the router, or to the router(mgmnt)?

I can do a netdr to see what traffic goes to it. It should not generate CPU load, but I thnk I will wait untill afterhours before I try that commnad.

- do NETDR:

-- debug netdr cap rx vlan 238

-- show netdr cap

Any repies are met with gratitude.

Cheers,

Tor Stefan

Reference

Input queue drops on 6500 vlan interface - https://supportforums.cisco.com/thread/2119895

Troubleshoot Input Queue Drops - http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic3

Show commands

xxx-k05-rs6#sh int vlan 238

Vlan238 is up, line protocol is up

Hardware is EtherSVI, address is 0023.040d.d1c0 (bia 0023.040d.d1c0)

Description: xxx.238.0_24 sak:P1381

Internet address is 161.4.238.1/24

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 46/255, rxload 159/255

Encapsulation ARPA, loopback not set

Keepalive not supported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 2d22h

Input queue: 0/75/29563976/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 898836000 bits/sec, 106595 packets/sec

5 minute output rate 183224000 bits/sec, 59158 packets/sec

L2 Switched: ucast: 3298256993 pkt, 1408022251821 bytes - mcast: 3460431 pkt, 476653445 bytes

L3 in Switched: ucast: 12851487771 pkt, 15530704413500 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 7375928081 pkt, 2467778470794 bytes mcast: 0 pkt, 0 bytes

16127281689 packets input, 17567277881823 bytes, 0 no buffer

Received 5080351 broadcasts (0 IP multicasts)

0 runts, 0 giants, 34740 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

7536851172 packets output, 2576927516962 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

-k05-rs6#sh int vlan 238 switching

Vlan238 servernett_161.4.238.0_24 sak:P1381

Throttle count 36171

Drops RP 30935411 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out

Other Process 380704913 34246329102 0 0

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

IP Process 1346950495 340903528931 160716715 18960732601

Cache misses 0

Fast 22225740 2814363123 87068544 28190395977

Auton/SSE 3319146968178 3623406840541663 1780517499010 852526331293699

DEC MOP Process 0 0 169820 21906780

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

ARP Process 1050722763 63043389126 119702916 13406726592

Cache misses 0

Fast 0 0 0 0

Auton/SSE 0 0 0 0

rsimoni · ‎06-18-2012

Hi Tor Stefan,

it seems indeed host with IP 161.4.238.134 and MAC 00.26.55.7F.66.C0 with is sending lots of ARP requests for all (or lots of) hosts in vlan 238. In the Target Protocl Address we can see lots of hosts (A104EEDE A104EEE5 etc).

Not sure if this is a malfunctioning, for sure it is the root cause.

You need to check what that server is doing... as a start check its arp table and see why it keeps sending those requests.

Meanwhile you could enlarge the input queue from the default 75 packets to something bigger (2000 for instance) or also you could configure the arp policer (sup720 feature) if things get worse.

Riccardo

View solution in original post

rsimoni · ‎06-18-2012

Hi,

It was a bit unclear to me in the thread "Input queue drops on 6500 vlan interface" if RP drops was through the router, or to the router(mgmnt)?

The counter indicates what is being dropped at RP level and does not diffirentiate from traffic going TO or THROUGH the router. In ideal conditions no transit traffic is supposed to go to the RP in order to be forwarded as it should be handled at hardware level by the PFC or DFC if present. So in this condition all the input drops happen for traffic destined TO the RP iteself (management traffic, protocols etc.). However in your outputs we can already see that IP process is forwarding some traffic. This should not happen as MLS CEF forwarding should occur instead (hardware level as I wrote above).

It would seem that traffic entering in vlan 238 (part of it at least) is sent to the RP to be forwarded by IP Process (not even in software CEF). NETDR will tell you exactly what is being 'punted' (this is the name of this operation).

Go for it, it is extremely safe as it will not add any load to the CPU as it is just a circular buffer.

Also, go and check the CPU utilization as you will likely see some interrupts utilization.

sh proc cpu sorted | e 0.00

It would also be interesting checking the following:

sh tcam int vlan 238 acl in ip det

and

sh mls cef exception status

Post these outputs (including netdr) for us to check.

Riccardo

torstefan · ‎06-18-2012

It seems to me that 00.26.55.7F.66.C0, which is a server I have now located, is generating loads of ARP (protocol 0806)

broadcasts.

Its a server which runs HP Insight Manager..

Could it be a mailfunc. nic?

But, here follows the different commands.

-k05-rs6#sh proc cpu sorted | e 0.00

CPU utilization for five seconds: 9%/3%; one minute: 8%; five minutes: 8%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

12 5063354361128506211 448 1.59% 0.88% 0.81% 0 ARP Input

538 5871670563215955670 0 1.51% 0.53% 0.52% 0 Port manager per

321 6192236963704860394 0 0.87% 0.43% 0.44% 0 IP Input

52 75552492 2614437 28898 0.79% 0.09% 0.06% 0 Per-minute Jobs

415 51906936 169324853 306 0.23% 0.06% 0.05% 0 CEF: IPv4 proces

250 262014368 847263604 309 0.23% 0.20% 0.19% 0 DHCPD Receive

358 55731092 59900016 930 0.15% 0.07% 0.06% 0 QOS Stats Gather

388 81079936 92758699 874 0.07% 0.09% 0.09% 0 IPC LC Message H

594 8327071482834682139 0 0.07% 0.21% 0.19% 0 SNMP ENGINE

315 31229984 350667956 89 0.07% 0.03% 0.01% 0 CDP Protocol

84 36948940 80829033 457 0.07% 0.05% 0.05% 0 ARP HA

-k05-rs6#sh mls cef exception status

Current IPv4 FIB exception state = FALSE

Current IPv6 FIB exception state = FALSE

Current MPLS FIB exception state = FALSE

-k05-rs6#sh tcam int vlan 238 acl in ip det

* Global Defaults shared

-------------------------------------------------------------------------------------------------------------------

DPort - Destination Port SPort - Source Port TCP-F - U -URG Pro - Protocol

I - Inverted LOU TOS - TOS Value - A -ACK rtr - Router

MRFM - M -MPLS Packet TN - T -Tcp Control - P -PSH COD - C -Bank Care Flag

- R -Recirc. Flag - N -Non-cachable - R -RST - I -OrdIndep. Flag

- F -Fragment Flag CAP - Capture Flag - S -SYN - D -Dynamic Flag

- M -More Fragments F-P - FlowMask-Prior. - F -FIN T - V(Value)/M(Mask)/R(Result)

X - XTAG (*) - Bank Priority

-------------------------------------------------------------------------------------------------------------------

Interface: 238 label: 1 lookup_type: 0

protocol: IP packet-type: 0

+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+

|T|Index| Dest Ip Addr | Source Ip Addr| DPort | SPort | TCP-F |Pro|MRFM|X|TOS|TN|COD|F-P|

+-+-----+---------------+---------------+---------------+---------------+-------+---+----+-+---+--+---+---+

Entries from Bank 0

V 18396 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- --- 0-0

M 18404 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0

R rslt: L3_DENY_RESULT rtr_rslt: L3_DENY_RESULT hit_cnt=0

Entries from Bank 1

V 36259 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- C-- 0-0 <-

M 36260 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0 <-

R rslt: PERMIT_RESULT (*) rtr_rslt: PERMIT_RESULT (*) hit_cnt=292182389 <-

V 36828 0.0.0.0 0.0.0.0 P=0 P=0 ------ 0 ---- 0 0 -- --- 0-0

M 36836 0.0.0.0 0.0.0.0 0 0 ------ 0 ---- 0 0

R rslt: L3_DENY_RESULT (*) rtr_rslt: L3_DENY_RESULT (*) hit_cnt=17

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

E0820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE4319

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE9A 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

E8820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE7B19

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE99 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

F0820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE3319

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE92 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

F8820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE0B19

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE96 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

00820400 00EE0000 0B5A0000 40080000 E0000040 0E000008 00000018 40EE0343

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.98.C8, protocol 0806

layer 3 data: 00010800 06040001 0026557F 98C8A104 EE920000 00000000

A104EE86 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

08820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE5B16

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE98 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

10820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE1316

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE8B 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

18820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE2B16

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE97 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

20820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE8316

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE94 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

28820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EEBB16

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE8C 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

30820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EEF316

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE79 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

38820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EECB16

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE93 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

48820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE1B13

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE90 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

50820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE5313

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE91 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

58820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE6B13

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE8E 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5C(2908), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

68820400 00EE0000 0B5C0000 40080000 E0000550 0E000008 00000008 40EEB9E5

mistral hdr: req_token 0x0(0), src_index 0xB5C(2908), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.22.19.8F.36.0D, protocol 0806

layer 3 data: 00010800 06040001 0022198F 360DA104 EE8E0000 00000000

A104EE86 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

70820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EEB313

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE89 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

78820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE8B13

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EE63 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

90820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE931C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEEB 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

98820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EEAB1C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEE8 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

A0820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE031C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEE9 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

A8820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE3B1C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEE7 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

B0820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE731C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEE0 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

B8820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EE4B1C

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEDE 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

------- dump of incoming inband packet -------

interface Vl238, routine mistral_process_rx_packet_inlin, timestamp 00:03:47

dbus info: src_vlan 0xEE(238), src_indx 0xB5A(2906), len 0x40(64)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x40EE(16622)

D8820400 00EE0000 0B5A0100 40080000 E0000520 02000008 00000008 40EEEB19

mistral hdr: req_token 0x0(0), src_index 0xB5A(2906), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xEE(238)

destmac FF.FF.FF.FF.FF.FF, srcmac 00.26.55.7F.66.C0, protocol 0806

layer 3 data: 00010800 06040001 0026557F 66C0A104 EE860000 00000000

A104EEE5 00000000 00000000 00000000 00000000 00000000

00000000 00000000 0000FFFF

rsimoni · ‎06-18-2012

Hi Tor Stefan,

it seems indeed host with IP 161.4.238.134 and MAC 00.26.55.7F.66.C0 with is sending lots of ARP requests for all (or lots of) hosts in vlan 238. In the Target Protocl Address we can see lots of hosts (A104EEDE A104EEE5 etc).

Not sure if this is a malfunctioning, for sure it is the root cause.

You need to check what that server is doing... as a start check its arp table and see why it keeps sending those requests.

Meanwhile you could enlarge the input queue from the default 75 packets to something bigger (2000 for instance) or also you could configure the arp policer (sup720 feature) if things get worse.

Riccardo

torstefan · ‎06-19-2012

Thank you for your answer. I also liked the little brainteaser. I had to find out where you got the host IP from. I don't work that much (yet) with hex, but I do have an advanced calculator in my Linux

rsimoni · ‎06-19-2012

indeed you can get lots of interesting info by exploring the captured data comparing it with an ARP frame

http://ipv6.com/articles/general/Address-Resolution-Protocol.htm

you see the protocol (IP), the type of ARP (request), the sender, the target etc,