Input queue drops on 6500 vlan interface

[resbanknet]

Good morning.

Can someone please explain for a Vlan interface would be dropping packets on the input queue?  Please refer to the drops/flushes below.  This is from a 6500 with a Sup720, there are a number of vlans on it. This 6500 and it's HSRP partner are exhibiting the same symptoms on all the vlans I bothered to check.  This particular vlan is quite lightly used, there are only about fifteen user PC's (each with 100 Mb interfaces) on it.

There is a bit of information on input queue drops on Cisco, but this is focussed on physical interfaces where I can understand some packets being dropped.  I would think that Vlan interfaces would have different issues.

I note the "no buffer" errors as well, that also concerns me, especially as that counter is quite close to the "flushes".

Thanks

_____________________________________________________

Vlan123 is up, line protocol is up
  Hardware is EtherSVI, address is 00d0.04fd.6000 (bia 00d0.04fd.6000)
  Description: Vlan123
  Internet address is 10.123.123.7/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1w5d
  Input queue: 0/75/48016/4665 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 172000 bits/sec, 35 packets/sec
  L2 Switched: ucast: 873 pkt, 96349 bytes - mcast: 5721320 pkt, 5631745097 byte
s
  L3 in Switched: ucast: 12 pkt, 888 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 43032751 pkt, 26311968968 bytes mcast: 43650871 pkt, 5
7721556831 bytes
     2725081 packets input, 213967275 bytes, 4594 no buffer
     Received 2706655 broadcasts (1037178 IP multicasts)
     0 runts, 0 giants, 6115 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     47543646 packets output, 26462353272 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out

interface Vlan123
description Vlan123
ip address 10.123.123.7 255.255.255.0
ip helper-address 10.123.120.5
ip helper-address 10.123.120.5
ip directed-broadcast 109
ip pim sparse-dense-mode
standby 10.123.123.1
standby priority 90
no mop enabled
end

[nkarpysh]

Hello,

Usually this can be seen when traffic arrving to SVI should be sent for CPU processing. If that traffic dropped by CPU - you will see the drops on VLAN input. Also if queue is full with these packets you will see flushes and no buffers.

You can check "show int vlan 123 switching" to see if you similar RP drops there.

So to mitigate this you need to understand what packets are punted to CPU, why those are punted and then eliminate of these packets.

You can do it with follwoing methods:

- SPAN VLAN 123 or particular VLAN ports on ingreass and watch those packest with wireshark to see what is wrong there

- do NETDR:

-- debug netdr cap rx vlan 123

-- show netdr cap

it will show you wich packets are punted to CPU ingressing VLAN 123 - it is safe to use as it is used for TS of High CPU issues.

Let me know if any further help needed.

Nik

[resbanknet]

Thanks for your reply.  I am attaching the output of the "show switching" and refresh of the "show interface" below, I am not sure if anything draws your attention from the output.

About that netdr command, will that really only show the packets sent from the vlan input queue to the CPU?  The command just looks like it will show all input packets?  I would like to run this command but it is definitely after-hours stuff.

Thanks

__________________________________________________________

C6500XX#sh int vlan 123 switching
Vlan123 Vlan 123
          Throttle count       6162
        Drops         RP      48267         SP          0
  SPD Flushes       Fast       4692        SSE          0
  SPD Aggress       Fast          0
SPD Priority     Inputs    2505229      Drops          0

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
        Other    Process       2420     170246      54787    6136144
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
           IP    Process    4805278  387346451    7451746  895606520
            Cache misses          0
                    Fast      55640    5410302       5881    5170976
               Auton/SSE       1980     346560   73539176 57912210832
      DEC MOP    Process        136      10472        134      17286
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
          ARP    Process      60028    3601680      21818    2443616
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

C6500XX#sh int vlan 123
Vlan123 is up, line protocol is up
  Hardware is EtherSVI, address is 00d0.04fd.6000 (bia 00d0.04fd.6000)
  Description: Vlan 123
  Internet address is 10.123.123.7/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1w6d
  Input queue: 6/75/48267/4692 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 153000 bits/sec, 15 packets/sec
  L2 Switched: ucast: 1019 pkt, 109258 bytes - mcast: 5735619 pkt, 5633102928 bytes
  L3 in Switched: ucast: 12 pkt, 888 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 44588005 pkt, 28158130332 bytes mcast: 43651402 pkt, 57721621480 bytes
     2752070 packets input, 216231107 bytes, 4609 no buffer
     Received 2733494 broadcasts (1046051 IP multicasts)
     0 runts, 0 giants, 6162 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     49137624 packets output, 28305037685 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out

[nkarpysh]

Watch RP drops - those macth your input drops:

   Drops         RP      48267         SP          0

Input queue: 6/75/48267/4692 (size/max/drops/flushes); Total output drops: 0

SO indeed some traffic punted to CPU is dropped there. NETDR command will catch all packets going to cpu - not usually that much as most of the traffic handled in HW. And that is harmless as give output by pages and does not cause any CPU spike.

Please also note you need to do that command when counters are growing as it catchs packets in real time.

Nik