Hello guys
I'm writing because I have observed something strange on one of the 9300 stacked switches. I thought I had found a solution by creating a port channel within the same NIM module instead of spanning across modules. In that case, we experienced no packet loss. We had 4-5 PCs connected with Iperf3 running at almost 1 gigabit for 20 minutes, and the output loss in port channel 98 was minimal. However, when we tried the same on the other switch, there was packet loss even with just a single PC. I simply don't understand this. We're running version 17.06.05 on both switches.
The setup is as follows: 2 sessions with 3 hosts in each—no packet loss in one port channel, but there is in the other. The strange thing is that I've tried replacing the C9300 24T, but the output remains the same. As I mentioned earlier, the monitor session has been set to "rx" since it seems to be the only potential solution based on what I've seen in Wireshark in terms of minimizing the number of packets. The purpose of this task is to monitor all ports, and we can't use 10-gigabit because the Siem devices only support 1 gigabit.
1: Can you confirm that SPAN is not meant to monitor an entire switch but is designed for troubleshooting specific issues?
2: I can see that there are also some tap solutions, but again, I couldn't find anything that supports 24 copper connections at 1 gigabit; the ones I found were for 100 megabit.
Iperf configuration:
UDP-TEST:> iperf3 -c 192.168.40.1 -p 7575 -u -t 1200 -b 1000M
TCP-TEST:> iperf3 -c 192.168.40.1 -p 7575 -P 8 -t 1200 -b 1000M
We also tried with different frame sizes 64-1514 bytes and jumbo frames same issue with switch 2
Switch configuration:
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
A - formed by Auto LAG
Number of channel-groups in use: 2
Number of aggregators: 2
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
98 Po98(SU) LACP Te1/1/3(P) Te1/1/4(P)
99 Po99(SU) LACP Te2/1/3(P) Te2/1/4(P)
98 = Switch 1
99 = Switch 2
Session 1
Type : Local Session
Source Ports :
RX Only : Gi1/0/1-24,Te1/1/1-2,Te1/1/5-8
Destination Ports : Po98
Encapsulation : Replicate
Ingress : Disabled
Session 2
---------
Type : Local Session
Source Ports :
Rx Only : Gi2/0/16-17,Gi2/0/20-21,Gi2/0/23-24
Destination Ports : Po99
Encapsulation : Replicate
Ingress : Disabled
In session 2, which is failing, I have only selected the source ports where we have connected the PC’s and the iperf server.
Here is output drops with two PC’s running
test-switch#show interface po99 | i line|escription|bits|drops
Port-channel99 is up, line protocol is down (monitoring)
Description: *** Monitor Session 2 IDS ***
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 871992
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 247563000 bits/sec, 21816 packets/sec
0 unknown protocol drops
PC1 - iperf test
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 6] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 8] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 10] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 12] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 14] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 16] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[ 18] 70.01-71.01 sec 6.62 MBytes 55.5 Mbits/sec
[SUM] 70.01-71.01 sec 53.0 MBytes 444 Mbits/sec
PC2
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 70.01-71.01 sec 14.1 MBytes 118 Mbits/sec
[ 6] 70.01-71.01 sec 14.0 MBytes 118 Mbits/sec
[ 8] 70.01-71.01 sec 14.0 MBytes 118 Mbits/sec
[ 10] 70.01-71.01 sec 14.1 MBytes 114 Mbits/sec
[ 12] 70.01-71.01 sec 13.9 MBytes 118 Mbits/sec
[ 14] 70.01-71.01 sec 14.1 MBytes 118 Mbits/sec
[ 16] 70.01-71.01 sec 14.1 MBytes 118 Mbits/sec
[ 18] 70.01-71.01 sec 14.0 MBytes 118 Mbits/sec
[SUM] 70.01-71.01 sec 53.0 MBytes 939 Mbits/sec
NB: I also tried to change the port channel to use the following algorithm (src-mixed-ip-port) even though the switches only handle layer 2 traffic.. it seems it helped a bit but it is still not good. any suggestions? I have also tried to make a port channel with 4 interfaces in bundle - and I still got packet loss...
So I'm pretty confused....
/Jeppe
