Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2684
Views
10
Helpful
9
Replies

Frames output discards on NEXUS FEX PORTs

from88
Level 4
Level 4

‎07-23-2019 04:03 AM - edited ‎07-23-2019 04:04 AM

Hello,

 

I've got a DC topology where i get ingress traffic from 3x10GB lagged interface which are egress to FEX port (2x10gb). And on that 2x10GB ports i got lots of output discards counting ON. I got no specified queening  configured, but the couter of dropped bytes is scary.

 

show queuing interface ethernet 1/23 
+-------------------------------------------------------------------+
| Tx Pkts | 6489633480490| 395439| 11794585|
| Tx Byts |8481580517561871| 600980859| 2133802229|
| Dropped Pkts | 7303751537| 0| 0|
| Dropped Byts | 10498912379407| 0| 0|
| Q Depth Byts | 32240| 0| 0|

Is there anything you could recommend ? Create a qos with buffers ? Etc ? I dont know why it's happening since the ingress and egress interfaces is the same capacity (10gb)

 

Device: cisco Nexus9000 C9372PX

 

Thank you for any input.

Labels:
0 Helpful
9 Replies 9

Remi Astruc
Level 1
Level 1

‎07-23-2019 11:39 PM

Hi @from88 ,

Can you paste the full output of command show queuing interface ethernet 1/23?

Also show interface e1/23,  show hardware rate-limit module 1 and show policy-map interface control-plane.

 

Remi Astruc

0 Helpful

from88
Level 4
Level 4
In response to Remi Astruc

‎07-24-2019 12:17 AM

hello,

 

Here you are:

 

show interface e1/23
Ethernet1/23 is up
admin state is up, Dedicated Interface
  Belongs to Po25
  Hardware: 1000/10000 Ethernet, address: 0062.ec1a.191c (bia 0062.ec1a.191c)
  Description: ITXFEX125
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 32/255, rxload 39/255
  Encapsulation ARPA, medium is broadcast
  Port mode is fex-fabric
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Auto-Negotiation is turned on, FEC mode is Auto
  Input flow-control is off, output flow-control is off
  Auto-mdix is turned off
  Rate mode is dedicated
  Switchport monitor is off 
  EtherType is 0x8100 
  EEE (efficient-ethernet) : n/a
  Last link flapped 71week(s) 0day(s)
  Last clearing of "show interface" counters 20:37:01
  0 interface resets
  30 seconds input rate 1560567800 bits/sec, 150573 packets/sec
  30 seconds output rate 1273727960 bits/sec, 125790 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 2.60 Gbps, 229.60 Kpps; output rate 2.17 Gbps, 193.72 Kpps
  RX
    16215032495 unicast packets  24218 multicast packets  0 broadcast packets
    16215056713 input packets  20952139531262 bytes
    13275675582 jumbo packets  0 storm suppression packets
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    15306294432 unicast packets  768894 multicast packets  2584 broadcast packets
    15307065910 output packets  20140093661160 bytes
    12780926079 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble  3360277 output discard
    0 Tx pause

show hardware rate-limit module 1

Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters

Module: 1
  R-L Class           Config           Allowed         Dropped            Total
 +------------------+--------+---------------+---------------+-----------------+
  L3 glean                 100       183852281        24073009         207925290
  L3 mcast loc-grp        3000               0               0                 0
  access-list-log          100               0               0                 0
  bfd                    10000               0               0                 0
  exception                 50               0               0                 0
  fex                     3000      1055027064               0        1055027064
  span                      50               0               0                 0
  dpss                    6400               0               0                 0
  sflow                  40000               0               0                 0


show policy-map interface control-plane
Control Plane

  Service-policy  input: copp-system-p-policy-strict

    class-map copp-system-p-class-l3uc-data (match-any)
      match exception glean
      set cos 1
      police cir 250 pps , bc 32 packets 
      module 1 :
        transmitted 765315655 packets;
        dropped 6366861 packets;

    class-map copp-system-p-class-critical (match-any)
      match access-group name copp-system-p-acl-bgp
      match access-group name copp-system-p-acl-rip
      match access-group name copp-system-p-acl-vpc
      match access-group name copp-system-p-acl-bgp6
      match access-group name copp-system-p-acl-ospf
      match access-group name copp-system-p-acl-rip6
      match access-group name copp-system-p-acl-eigrp
      match access-group name copp-system-p-acl-ospf6
      match access-group name copp-system-p-acl-eigrp6
      match access-group name copp-system-p-acl-auto-rp
      match access-group name copp-system-p-acl-mac-l3-isis
      set cos 7
      police cir 19000 pps , bc 128 packets 
      module 1 :
        transmitted 42048305 packets;
        dropped 0 packets;

    class-map copp-system-p-class-important (match-any)
      match access-group name copp-system-p-acl-hsrp
      match access-group name copp-system-p-acl-vrrp
      match access-group name copp-system-p-acl-hsrp6
      match access-group name copp-system-p-acl-vrrp6
      match access-group name copp-system-p-acl-mac-lldp
      set cos 6
      police cir 3000 pps , bc 128 packets 
      module 1 :
        transmitted 229270471 packets;
        dropped 0 packets;

    class-map copp-system-p-class-openflow (match-any)
      match access-group name copp-system-p-acl-openflow
      set cos 5
      police cir 2000 pps , bc 32 packets 
      module 1 :
        transmitted 118 packets;
        dropped 0 packets;

    class-map copp-system-p-class-multicast-router (match-any)
      match access-group name copp-system-p-acl-pim
      match access-group name copp-system-p-acl-msdp
      match access-group name copp-system-p-acl-pim6
      match access-group name copp-system-p-acl-pim-reg
      match access-group name copp-system-p-acl-pim6-reg
      match access-group name copp-system-p-acl-pim-mdt-join
      set cos 6
      police cir 3000 pps , bc 128 packets 
      module 1 :
        transmitted 12 packets;
        dropped 0 packets;

    class-map copp-system-p-class-multicast-host (match-any)
      match access-group name copp-system-p-acl-mld
      set cos 1
      police cir 2000 pps , bc 128 packets 
      module 1 :
        transmitted 911634 packets;
        dropped 0 packets;

    class-map copp-system-p-class-l3mc-data (match-any)
      match exception multicast rpf-failure
      match exception multicast dest-miss
      set cos 1
      police cir 3000 pps , bc 32 packets 
      module 1 :
        transmitted 0 packets;
        dropped 0 packets;

    class-map copp-system-p-class-normal (match-any)
      match access-group name copp-system-p-acl-mac-dot1x
      match protocol arp
      set cos 1
      police cir 1500 pps , bc 32 packets 
      module 1 :
        transmitted 2783172463 packets;
        dropped 96739 packets;

    class-map copp-system-p-class-ndp (match-any)
      match access-group name copp-system-p-acl-ndp
      set cos 6
      police cir 1500 pps , bc 32 packets 
      module 1 :
        transmitted 563115 packets;
        dropped 0 packets;

    class-map copp-system-p-class-normal-dhcp (match-any)
      match access-group name copp-system-p-acl-dhcp
      match access-group name copp-system-p-acl-dhcp6
      set cos 1
      police cir 300 pps , bc 32 packets 
      module 1 :
        transmitted 22739 packets;
        dropped 0 packets;

    class-map copp-system-p-class-normal-dhcp-relay-response (match-any)
      match access-group name copp-system-p-acl-dhcp-relay-response
      match access-group name copp-system-p-acl-dhcp6-relay-response
      set cos 1
      police cir 400 pps , bc 64 packets 
      module 1 :
        transmitted 115514 packets;
        dropped 0 packets;

    class-map copp-system-p-class-normal-igmp (match-any)
      match access-group name copp-system-p-acl-igmp
      set cos 3
      police cir 6000 pps , bc 64 packets 
      module 1 :
        transmitted 220416 packets;
        dropped 0 packets;

    class-map copp-system-p-class-redirect (match-any)
      match access-group name copp-system-p-acl-ptp
      set cos 1
      police cir 1500 pps , bc 32 packets 
      module 1 :
        transmitted 0 packets;
        dropped 0 packets;

    class-map copp-system-p-class-exception (match-any)
      match exception ip option
      match exception ip icmp unreachable
      match exception ipv6 option
      match exception ipv6 icmp unreachable
      set cos 1
      police cir 50 pps , bc 32 packets 
      module 1 :
        transmitted 68 packets;
        dropped 21 packets;

    class-map copp-system-p-class-exception-diag (match-any)
      match exception ttl-failure
      match exception mtu-failure
      set cos 1
      police cir 50 pps , bc 32 packets 
      module 1 :
        transmitted 4009922 packets;
        dropped 53 packets;

    class-map copp-system-p-class-management (match-any)
      match access-group name copp-system-p-acl-ftp
      match access-group name copp-system-p-acl-ntp
      match access-group name copp-system-p-acl-ssh
      match access-group name copp-system-p-acl-http
      match access-group name copp-system-p-acl-ntp6
      match access-group name copp-system-p-acl-sftp
      match access-group name copp-system-p-acl-snmp
      match access-group name copp-system-p-acl-ssh6
      match access-group name copp-system-p-acl-tftp
      match access-group name copp-system-p-acl-https
      match access-group name copp-system-p-acl-snmp6
      match access-group name copp-system-p-acl-tftp6
      match access-group name copp-system-p-acl-radius
      match access-group name copp-system-p-acl-tacacs
      match access-group name copp-system-p-acl-telnet
      match access-group name copp-system-p-acl-radius6
      match access-group name copp-system-p-acl-tacacs6
      match access-group name copp-system-p-acl-telnet6
      set cos 2
      police cir 3000 pps , bc 32 packets 
      module 1 :
        transmitted 655292718 packets;
        dropped 0 packets;

    class-map copp-system-p-class-monitoring (match-any)
      match access-group name copp-system-p-acl-icmp
      match access-group name copp-system-p-acl-icmp6
      match access-group name copp-system-p-acl-traceroute
      set cos 1
      police cir 300 pps , bc 128 packets 
      module 1 :
        transmitted 101610314 packets;
        dropped 0 packets;

    class-map copp-system-p-class-l2-unpoliced (match-any)
      match access-group name copp-system-p-acl-mac-stp
      match access-group name copp-system-p-acl-mac-lacp
      match access-group name copp-system-p-acl-mac-cfsoe
      match access-group name copp-system-p-acl-mac-sdp-srp
      match access-group name copp-system-p-acl-mac-l2-tunnel
      match access-group name copp-system-p-acl-mac-cdp-udld-vtp
      set cos 7
      police cir 20000 pps , bc 8192 packets 
      module 1 :
        transmitted 2472044401 packets;
        dropped 0 packets;

    class-map copp-system-p-class-undesirable (match-any)
      match access-group name copp-system-p-acl-undesirable
      match exception multicast sg-rpf-failure
      set cos 0
      police cir 15 pps , bc 32 packets 
      module 1 :
        transmitted 2485 packets;
        dropped 0 packets;

    class-map copp-system-p-class-fcoe (match-any)
      match access-group name copp-system-p-acl-mac-fcoe
      set cos 6
      police cir 1500 pps , bc 128 packets 
      module 1 :
        transmitted 0 packets;
        dropped 0 packets;

    class-map copp-system-p-class-nat-flow (match-any)
      match exception nat-flow
      set cos 7
      police cir 100 pps , bc 64 packets 
      module 1 :
        transmitted 0 packets;
        dropped 0 packets;

    class-map copp-system-p-class-l2-default (match-any)
      match access-group name copp-system-p-acl-mac-undesirable
      set cos 0
      police cir 50 pps , bc 32 packets 
      module 1 :
        transmitted 30397883 packets;
        dropped 6425137 packets;

    class-map class-default (match-any)
      set cos 0
      police cir 50 pps , bc 32 packets 
      module 1 :
        transmitted 22048545 packets;
        dropped 1885728 packets;

do you see something suspicious ?

 

Thank you

0 Helpful

Remi Astruc
Level 1
Level 1
In response to from88

‎07-24-2019 01:39 AM

Hi,

You missed also to paste the full output of command show queuing interface ethernet 1/23

 

Remi Astruc

0 Helpful

from88
Level 4
Level 4
In response to Remi Astruc

‎07-24-2019 03:55 AM - edited ‎07-24-2019 04:03 AM

sorry, this is the output:

 

show queuing interface ethernet 1/23

slot  1
=======


Egress Queuing for Ethernet1/23 [System]
------------------------------------------------------------------------------
QoS-Group# Bandwidth% PrioLevel                Shape                   QLimit
                                   Min          Max        Units   
------------------------------------------------------------------------------
      3             -         1           -            -     -            6(D)
      2             0         -           -            -     -            6(D)
      1             0         -           -            -     -            6(D)
      0           100         -           -            -     -            6(D)
+-------------------------------------------------------------------+
|                              QOS GROUP 0                          |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |   9528613680752|          641844|          202350|
|        Tx Byts |12613165589837630|       932469022|        69962346|
|   Dropped Pkts |      3108032358|               0|             782|
|   Dropped Byts |   4522891928871|               0|         1184204|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+
|                              QOS GROUP 1                          |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |               0|               0|               0|
|        Tx Byts |               0|               0|               0|
|   Dropped Pkts |               0|               0|               0|
|   Dropped Byts |               0|               0|               0|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+
|                              QOS GROUP 2                          |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |               0|               0|               0|
|        Tx Byts |               0|               0|               0|
|   Dropped Pkts |               0|               0|               0|
|   Dropped Byts |               0|               0|               0|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+
|                              QOS GROUP 3                          |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |               0|               0|               0|
|        Tx Byts |               0|               0|               0|
|   Dropped Pkts |               0|               0|               0|
|   Dropped Byts |               0|               0|               0|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+
|                      CONTROL QOS GROUP                            |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |        14880291|               0|       358993147|
|        Tx Byts |      9107829873|               0|     26578915538|
|   Dropped Pkts |               1|               0|               0|
|   Dropped Byts |             642|               0|               0|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+
|                         SPAN QOS GROUP                            |
+-------------------------------------------------------------------+
|                |  Unicast       | OOBFC Unicast  |  Multicast     |
+-------------------------------------------------------------------+
|        Tx Pkts |               0|               0|               0|
|        Tx Byts |               0|               0|               0|
|   Dropped Pkts |               0|               0|               0|
|   Dropped Byts |               0|               0|               0|
|   Q Depth Byts |               0|               0|               0|
+-------------------------------------------------------------------+

Port Egress Statistics
--------------------------------------------------------
WRED Drop Pkts                              0
EOQ(qos-group-0) Drop Pkts                   0



Ingress Queuing for Ethernet1/23
------------------------------------------------------------------
QoS-Group#                 Pause                        QLimit
           Buff Size       Pause Th      Resume Th      
------------------------------------------------------------------
      3              -            -            -           10(D)
      2              -            -            -           10(D)
      1              -            -            -           10(D)
      0              -            -            -           10(D)


Port Ingress Statistics
--------------------------------------------------------
Ingress MMU Drop Pkts                               0
Ingress MMU Drop Bytes                              0

PFC Statistics
----------------------------------------------------------------------------
TxPPP:                    0, RxPPP:                    0
----------------------------------------------------------------------------
 COS QOS Group        PG   TxPause   TxCount         RxPause         RxCount
   0         -         -  Inactive         0        Inactive               0
   1         -         -  Inactive         0        Inactive               0
   2         -         -  Inactive         0        Inactive               0
   3         -         -  Inactive         0        Inactive               0
   4         -         -  Inactive         0        Inactive               0
   5         -         -  Inactive         0        Inactive               0
   6         -         -  Inactive         0        Inactive               0
   7         -         -  Inactive         0        Inactive               0
----------------------------------------------------------------------------

Btw "Q Depth Byts" sometimes changes to 1040. Seems like this entry is dynamic one.

0 Helpful

Remi Astruc
Level 1
Level 1
In response to from88

‎07-24-2019 06:49 AM - edited ‎07-24-2019 06:52 AM

Hi,

The port supports a lot of traffic, and probably some bursty applications.

Although your input ports are also 10G, you can have output issues when several input ports egress the same port.

You should identify your bursty application and mitigate them. Also you can upgrade the port to a higher speed, or add ports in a Port-channel.

You can also try to change the Burst Profile with hardware qos ns-buffer-profile ultra-burst.

Finally if that cannot be enhanced, you will need to implement QoS to segregate some traffic explicitly.

Anyway it is worth opening a TAC case for deeper tshooting.

 

Remi Astruc

from88
Level 4
Level 4
In response to Remi Astruc

‎07-30-2019 06:47 AM

thank you, so much.

 

I want to try this: hardware qos ns-buffer-profile ultra-burst.

Maybe you know, is there are any disruption entering this command ? Do this require any reboot ,etc ?

 

Thank you

0 Helpful

Remi Astruc
Level 1
Level 1
In response to from88

‎07-30-2019 09:46 AM

Hello,

I don't really know.

I suggest you plan your maintenance window like if you would need a reboot after entering the command, just in case.

 

Remi Astruc

from88
Level 4
Level 4
In response to Remi Astruc

‎07-31-2019 12:49 AM

Thanks again,

Strange thing is that these discards mainly occurs on N9K-C9372-PX-E devices. We also have - N5K-C5548UP, N5K-C5672UP and on them there are no dicards. Could it be that N5K-C5548UP, N5K-C5672UP has bigger buffers than N9Ks ?

0 Helpful

from88
Level 4
Level 4
In response to from88

‎08-03-2019 04:56 AM

and maybe you can say - how to check the buffer interface has allocated ? to see if hardware burst profile ultra-burst helps.

 

Thank you!

0 Helpful
Customers Also Viewed These Support Documents

Review Cisco Networking for a $25 gift card