cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1902
Views
8
Helpful
33
Replies

Cisco 9200 packet loss when using 10G SFP

carl_townshend
Spotlight
Spotlight

Hi All

We are having some issues wherby when we plug in a 10G SFP int our 9200L-48P-4X switch we get severe performance loss, if we put in a 1G SFP the connection is fine.

We have tried multiple SFPs, different ports, different fibers, software upgrade etc and we will get lots of packet loss, this is backed up with some wireshark testing.

Software is on, 17.09.05 Cupertino

Any known issues here?

33 Replies 33

Joseph W. Doherty
Hall of Fame
Hall of Fame

Interesting.

I cannot comment on bugs, but how do you see the loss, interface packet drops?  Loss tied to utilization?

Years ago, I recall a posting on a similar case, when a switch uplink was changed from 100 Mbps to gig.

Hi Joseph

We see the loss from when we do throughput testing from the LAN and to the internet, we also use a product called Appneta pathview which also tells us there is loss, backed up with wireshark captures full of dup acks etc.

Not sure where we go with it, as we have ruled lots of things out, we even tried the same model switch, a new one and it was the same.

Would have this gig port be a potential bottleneck to the Internet vs. it being 10g?

If so, possibly slamming the Internet harder has exposed a transit issue not seen until higher burst rates are possible.

An interesting test, if switch supports shaping and policing, install 10g but shape egress at gig and possibly police ingress at gig too.  See how that compares to gig and 10g natively.

If results match native gig, increase rates to two gig.  If still no major loss, increase to 3 gig, etc.

The foregoing to try to see if the issue is the overall path has an issue with higher burst rates.

Hi Joseph

Its not a bottle neck, we have tried the tests to a server on the LAN running at 1G, the laptop we are doing the testing on is also running at 1G, so no bottlenecks

What port is the one being converted from gig to 10g, server<>9200L or 9200L<>9500?

You wrote earlier "throughput testing from the LAN and to the internet" and now "tests to a server on the LAN running at 1G, the laptop we are doing the testing on is also running at 1G", so @carl_townshend could you clarify what link is changed and what host(s) are then impacted.

As to my comment about "bottlenecks", it may have been misunderstood.  Moving from gig to 10g removes the gig "bottleneck" compared to 10g, which would normally be considered a "good thing".  But, that's not aways the case.

NICs should always be able to handle a single frame at line rate.  The device having that NIC may not be able to keep up with processing sustained frames/packets.  Just as a microburst can overrun a buffer, causing drops, while overall average utilization is low, it can do the same to frame/packet processing.

So, there are cases where increasing bandwidth causes increased drops.  It's counter intuitive, unless you understand increasing bandwidth also increases PPS.

If drops are being increased due to additional PPS demand, devices (at least Cisco switches and routers) will often show that somewhere in their stats, but not as egress drops.

If this issue is strictly tied to Internet, finding a too PPS stressed Internet device can be practically impossible (unless you're the ISP).

If problem can be seen internal to your network, you have a good chance finding it.

BTW, the PPS issue, decades ago, was much more common.  Which is why, back then, vendors, including Cisco, would often just provide a PPS for 1500 byte packets, but often not for smaller packets.  The reason being device could support wire-rate for that sized packets, but not for below some smaller size.

I'll also mention this issue is much more relevant to CPU based routers, rather than L3 switches, but even the latter can have issues in some rare cases.  (As such cases can be used for DoS attacks, I'll skip details.)

Helllo @carl_townshend ,

what is connected on the other end ? I mean what is the ISP CPE router you are connecting to ?

>> we do throughput testing from the LAN and to the internet

because the issue can be there on that side if it does not support full 10 Gbps link you might see losses on Catalyst side if flow control is enabled.

check with show interface and post it on the thread

Hope to help

Giuseppe

 

marce1000
Hall of Fame
Hall of Fame

 

 - I would recommend to use latest advisory : https://software.cisco.com/download/home/286320055/type/282046477/release/Dublin-17.12.4

 M.



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

M02@rt37
VIP
VIP

Hello @carl_townshend 

If you're using non-Cisco 10G SFP modules, that could definitely be a factor...

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

Leo Laohoo
Hall of Fame
Hall of Fame

What kind of "drops"?

Can we see the complete output to the command "sh interface <Ten>"?

Is this the "core" switch?

There are no drops on the interface, the drops are coming on the PC, shown in wireshark and basically slow speeds, packet loss shown on our monitoring system to the PC also.

this is an edge switch

Its connected like this , 9200L (access) > 10G  > 9500 (distribution)> 10G > 9600 (core)

How about any drops in the 10 Gbps ports?

10 Gbps optic means OM3/OM4.  What if the fibre plant is OM1/OM2?

siskum
Spotlight
Spotlight

Hi Carl,

I believe that when mismatch the port speed the packet loss occurs. You have already mentioned when 1G SFP connected to server no any packet loss happened (if we put in a 1G SFP the connection is fine.) Only packet losses occurred when 10G SFP interface connect with to the server port. In this case you have to rest the port (10G SFP) or set it to "auto" that 10G SFP in your Switch. Then both ports will negosiate what is the best speed to transmit and receive packets without any loss.

Sisira


@siskum wrote:

Hi Carl,

I believe that when mismatch the port speed the packet loss occurs. You have already mentioned when 1G SFP connected to server no any packet loss happened (if we put in a 1G SFP the connection is fine.) Only packet losses occurred when 10G SFP interface connect with to the server port. In this case you have to rest the port (10G SFP) or set it to "auto" that 10G SFP in your Switch. Then both ports will negosiate what is the best speed to transmit and receive packets without any loss.


@siskum , hmm, I do recall reading of rare (?) cases where the signal modulation rate is out of tolerance.  Is that the kind of speed mismatch you have in mind?

But if you're thinking port running one side at gig and the other side 10g, to the best of my knowledge, that doesn't work at all, for fiber or copper, not just poorly, like having a high loss rate (much like seen with copper duplex mismatches).  If that's what you have in mind, do you have any references for that, as I'm just not aware of such cases?