Re: Nexus 9K Span Port - Output Discards

njsanders1 · ‎07-16-2018

I have a couple of span ports configured for sending our northbound & southbound traffic to/from our firewalls over to our IDS sensors for examination. They are both set up as 1,000Mbps connections over copper media, and the span interfaces themselves are seeing no more than 300-350Mbps on the 5-minute average metrics. However, we see a substantial amount of output discards on both of the span ports. We do not see these discards on the source ports, incidentally. Our ISO is concerned that we are missing potential IDS data, given the large relative percentage of lost packets. Please advise as to possible causes AND solutions for this issue. Thank you.

Here's the configuration of the interfaces involved:

monitor session 1
description FW (Eth1/47)
source interface Ethernet1/47 both
destination interface Ethernet1/42
no shut

interface Ethernet1/47
description Fortigate 300C (dot1q)
switchport mode trunk
switchport trunk allowed vlan 30-31,101-110,199,210,631,742,746

interface Ethernet1/42
description IDS - SPAN
switchport monitor
spanning-tree port type edge
spanning-tree guard root

monitor session 2
description Rack Eth1/48 (vlan631,742,746)
source interface Ethernet1/48 both
destination interface Ethernet1/40
no shut

interface Ethernet1/48
description Rack 1208.xx.yy (dot1q)
switchport mode trunk
switchport trunk allowed vlan 631,742,746

interface Ethernet1/40
description IDS - SPAN
switchport monitor
spanning-tree port type edge
spanning-tree guard root

Here's how the ports are performing presently:

# sho int Eth1/40
Ethernet1/40 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: ecbd.1dxx.xxxx(bia ecbd.1dxx.xxxx)
Description: IDS - SPAN
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 45/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is on
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 01:10:32
Last clearing of "show interface" counters 00:09:30
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 176472352 bits/sec, 51078 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 174.48 Mbps, 51.01 Kpps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
28848548 unicast packets 3269 multicast packets 3123 broadcast packets
28854940 output packets 12317477510 bytes
1677744 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 97130 output discard
0 Tx pause

1# sho int Eth1/42
Ethernet1/42 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: ecbd.1dca.3d23 (bia ecbd.1dca.3d23)
Description: IDS (SPAN)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 80/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is on
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 01:10:40
Last clearing of "show interface" counters 00:09:50
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 315007448 bits/sec, 91864 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 306.93 Mbps, 89.78 Kpps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
52749881 unicast packets 2451 multicast packets 169389 broadcast packets
52921721 output packets 22221788123 bytes
3049369 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 499578 output discard
0 Tx pause

Digging a little deeper, I looked at the queuing statistics for these span ports as well, and here's what I found:

#sho queuing interfa eth1/40

slot 1
=======

Egress Queuing for Ethernet1/40 [System]
------------------------------------------------------------------------------
QoS-Group# Bandwidth% PrioLevel Shape QLimit
Min Max Units
------------------------------------------------------------------------------
3 - 1 - - - 6(D)
2 10 - - - - 6(D)
1 15 - - - - 6(D)
0 75 - - - - 6(D)
+-------------------------------------------------------------------+
| SPAN QOS GROUP |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Tx Pkts | 0| 0| 594838727|
| Tx Byts | 0| 0| 259183501529|
| Dropped Pkts | 0| 0| 2928005|
| Dropped Byts | 0| 0| 3595523409|
| Q Depth Byts | 0| 0| 0|
+-------------------------------------------------------------------+

#sho queuing interfa eth1/42

slot 1
=======

Egress Queuing for Ethernet1/42 [System]
------------------------------------------------------------------------------
QoS-Group# Bandwidth% PrioLevel Shape QLimit
Min Max Units
------------------------------------------------------------------------------
3 - 1 - - - 6(D)
2 10 - - - - 6(D)
1 15 - - - - 6(D)
0 75 - - - - 6(D)
+-------------------------------------------------------------------+
| SPAN QOS GROUP |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Tx Pkts | 0| 0| 1029464098|
| Tx Byts | 0| 0| 430548318583|
| Dropped Pkts | 0| 0| 17257907|
| Dropped Byts | 0| 0| 14240091239|
| Q Depth Byts | 0| 0| 832|
+-------------------------------------------------------------------+

Rajeshkumar Gatti · ‎07-26-2018

Hi,

Your span config is set to sniff in both the Rx and Tx direction at the source and aggregating it and sending it to the sniffer.

Also the traffic rate show by show interface output is an average over a 30sec period which will never show the microburst traffic. Its highly likely that at regular intervals (sub second ) the aggregate traffic will be exceeding the line rate causing output discards. You will never see this if you just look at 30 second interval.

This link about microburst detection can be helpful even though it is listed for catalyst-

https://www.cisco.com/c/en/us/support/docs/lan-switching/switched-port-analyzer-span/116260-technote-wireshark-00.html

-Raj

njsanders1 · ‎07-27-2018

I assumed that the bursts were occurring, but it seems odd to me that these bursts would be so severe that we would be dropping upwards of 10% of the packets from the SPAN ports, especially when the overall utilization is relatively low. Also interesting to me is the fact that, if we use 10G SPAN ports instead of 1G, we don't drop any packets at all -- I presume this is because the buffers are more robust on the 10G ports (even though the physical interfaces are the same).

I believe that I read somewhere that buffer allocations cannot be manipulated on SPAN ports. If that is the case, are there any suggestions on solving this problem? We have to keep these bidirectional, because the IDS to which we're mirroring the traffic can't work (oddly enough) with two separate unidirectional source ports. Of course, the SPAN ports themselves are unidirectional already. :)

Rajeshkumar Gatti · ‎07-31-2018

The overall utilization is never an absolute real time view of the traffic on the wire and can lead to underestimating the actual traffic especially when output discards are seen. 10gig line rate will dequeue packets at a much higher clocking rate and if lets say you are bursting at 1.1 gig/sec rate, then that will be easily dequeued on a 10gig port without using any buffers. Buffers typically only come into play when packets wait to be put on the wire. Both 10gig and 1 gig will use the same general port buffers.

The best solution as you have already found out is going to be using 10gig port.

If you change port buffers and create more depth for packets to stay in the switch longer, you may have reduced drops or no drops depending on the burst and is not a guaranteed solution.

maxwell.n · ‎08-19-2019

Hi There.

Did you ever resolve this issue? I have a similar problem.

Thanks

Neal