Packet Discard

[Rishav Garg]

We have a netapp storage on our Nexus and we are seeing alot of pks discarded on ingreess. I know that the Nexus does ingress queing so this seems to point to congestion on an egress port. How can I figure out what port is causing the problem or has the congestion?

I looked on Ciscos website but I can't find a good troubleshooting docutment.

qos-group 4
    q-size: 29760, HW MTU: 9216 (9216 configured)
    drop-type: drop, xon: 0, xoff: 29760
    Statistics:
        Pkts received over the port             : 5589330309
        Ucast pkts sent to the cross-bar        : 5504968963
        Mcast pkts sent to the cross-bar        : 86720
        Ucast pkts received from the cross-bar  : 0
        Pkts sent to the port                   : 162325
        Pkts discarded on ingress               : 72274627
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

[Marcin Latosiewicz]

To answer your question, you can monitor per interface registers for QoS group 4 to see what's going on. I'd suggest doing this with TAC. 

I will try to answer your concern though, you have QoS group 4 with a pretty small queue size. I'm going to go and assume it's NFS. 

Any larger burst of data might cause that buffer to fill up.

I'd check first whether you have this (if you running ESX) 

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016122

Secondly your QoS group 4 is a drop class. There are a couple of choices, change queue depth, change class to drop-less (edit, the options before are among others ... which would include traffic to be load balanced in a PC/vPC).

[Rishav Garg]

Thanks for your reply.

Yes, You are right, we are using NFS datastore on QOS group 4 which we have set to drop with queue limit 29960 KB auto assigned. 

Can you tell me how to check that how much queue or port buffer for particular qos group is used or free. so that accordingly we will increase as per our requirement.

One more thing we didn't set the queue depth for QOS-group 4 it assigns automatically. when we created QOS group 4. I think it borrows queue size from default queue size i.e 640 KB per port in 5548 UP model.

Am i right?

[Marcin Latosiewicz]

Indeed that the default allocation. But on N5k (unlike UCS) you can extend it. 

We could write up a short doc about tracking the free icell count etc but it requires quite specific platform knowledge AND if you have small bursts of data you might not be able to catch it reliably. 

[Rishav Garg]

Thanks again.

I know that we can extend it in N5K by command queue limit under policy map type network qos.

We have large amount burst of data in NFS in QOS group 4.

I want t know the command how to check queue used or free currently in Nexus 5500.

Why it is dicarding the ingress packets on particular QOS Group 4.

[Marcin Latosiewicz]

If you really want to go that way...

Check the QoS group mapping to class (you can get that info from QD).

Assuming carmel ASIC, you can dump, per carmel instance information about buffer usage.

Example for carmel ASIC 1, class 4 (QoS group 2 for me), ports 4 and 5 (eth1/13 and eth1/4) 

show hardware internal carmel asic 1 registers match .*STA.*frh4.*addr_[4-5].*

Again remember this dumps the state of registers at the moment you execute, it's very hard to find a short buffer usage spike. 

To get information about icells you can match registers based on. 

.*STA.*free_cel.*addr_[4-5]

It's as simple as rewriting the regexp to match what you're looking for. 

Again, not the simplest way :-)

[Rishav Garg]

Hi,

i did as you said. 

Firstly i find  QoS group mapping to class

C|PRI| WRR (WRR  )| in MTU B/cells|out MTU B/cells|SEL|ENA
0|off|01999( 6553)|     2240/   14|     2240/   14|  0|YES
1|off|00040(   64)|     9280/   58|     9280/   58|  1|YES
2| on|00040(   64)|     9280/   58|     9280/   58|  2|YES
3|off|00fff( 4095)|     1600/   10|     1600/   10|  3|YES
4|off|00fff( 4095)|     9280/   58|     9280/   58|  4|YES
5|off|00040(   64)|     1600/   10|     1600/   10|  5|YES
6|off|00666( 1638)|     9280/   58|     9280/   58|  6|YES
7|off|00040(   64)|     1600/   10|     1600/   10|  7|YES

after that i checked about icells usage for particular interface.

Slot 0 Carmel 0 register contents:
Register Name                                          | Offset   | Value
-------------------------------------------------------+----------+-----------
car_bm_STA_free_cell_addr_0                            | 0x502e4  | 0xff3
car_bm_STA_free_cell_addr_1                            | 0x522e4  | 0xff3
car_bm_STA_free_cell_addr_2                            | 0x542e4  | 0xff3
car_bm_STA_free_cell_addr_3                            | 0x562e4  | 0xff3
car_bm_STA_free_cell_addr_4                            | 0x582e4  | 0xff3
car_bm_STA_free_cell_addr_5                            | 0x5a2e4  | 0xff3
car_bm_STA_free_cell_addr_6                            | 0x5c2e4  | 0xff3
car_bm_STA_free_cell_addr_7                            | 0x5e2e4  | 0xff3

It showing me the value oxff3 which means 4081.

Now can you tell me here 0xff3 what does this mean? is this normal or something else?