Got packet drops when using Fiber Cable in Catalyst 9500

[SaintEvn]

Hi All,

We're using StackWise Virtual 9500 switch in a new deployment. Two firewall was running HA and two 9500 stacked together.

All my VLANs gateway are in my firewall and 9500 was only layer2 mode.

We’ve created VLAN interfaces in firewall and trunk port-channel between Firewall and C9500.

The issue is when we tried to ping from user to VLAN Gateways, some VLANs show many packet drops.

We are connecting the Firewall and switch using Fiber cable. When using fiber cable between Firewall and Switch,  when we tried to ping the VLANs gateway  , random VLAN show many packet drops.Not all VLANs show packet drops. If we have 5 VLANS , 3 VLANS working normal but at lease 2 random VLANs got packet drops.

We’ve tried changing another Fiber Cable, SFP transceiver, change load-balancing method of the port-channel, reboot the switch, reboot the firewall ,changing switch interface,check spanning tree, remove the link from port-channel and testing with only one link ,changing speed, duplex, but still no luck.

And after a long time troubleshooting, we also tried with RJ45/UTP Copper Cable and it’s working normal. As long as we use Copper cable, all VLANs working fine but as soon as we changed back to Fiber cable, random VLANs got packet drops.

We also tested with our old switch 3850 stacked and it's working fine for both Fiber and UTP cable.

Firewall side has only 1G fiber ports , so we are using 1G SFP to connect between Firewall and C9500 although it can support up to 25G.

Now we are planning to change the firmware version of the switch.Anyting else I can do or check on this , please suggest me .Very appreciate for any help.

Cisco IOS XE Software, Version 17.06.03

Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.6.3, RELEASE SOFTWARE (fc4)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2022 by Cisco Systems, Inc.

Compiled Wed 30-Mar-22 23:09 by mcpre

ROM: IOS-XE ROMMON

BOOTLDR: System Bootstrap, Version 17.6.1r, RELEASE SOFTWARE (P)

stackwise-virtual
domain 10
no ip routing

!
license boot level network-advantage addon dna-advantage
!

redundancy
mode sso
main-cpu
standby console enable
!
transceiver type all

interface TwentyFiveGigE1/0/45
!
interface TwentyFiveGigE1/0/46
stackwise-virtual dual-active-detection

interface TwentyFiveGigE1/0/47
stackwise-virtual link 1

!
interface TwentyFiveGigE1/0/48
stackwise-virtual link 1

interface TwentyFiveGigE2/0/46
stackwise-virtual dual-active-detection
!
interface TwentyFiveGigE2/0/47
stackwise-virtual link 1
!
interface TwentyFiveGigE2/0/48
stackwise-virtual link 1

[Reza Sharifi]

Hi,

When using fiber, try to set the speed and duplex on both sides of the connection and test again. So, in your case, it will be 1Gig.

HTH

[SaintEvn]

Hi reza,

Thanks for the suggestion .We already tried changing speed >> 1000 and duplex full but still not working

[Reza Sharifi]

Hi,

Ok. For testing, what if you only use two links (no portchannels) connecting the firewalls to the 9500s?

HTH

[aramireztrv]

Hello SaintEvn,

Were you able to solve your problem? We have the same problem, packet loss between a Fortigate HA cluster connected to a Cisco 9500 Stack via 1Gbps SFP on Twe Interfaces. (Migration from a Cisco 6500) If we leave it connected to the C6500, no issues... (andt there is NO link aggregation between firewall and switch)

Thanks

[QuentinS]

Hi, we are having the same issue as well with a stackwise of 9500-24Y24C and a cluster of Forti600E after upgrading from 16.12.04 to 17.06.04 and 17.06.05 also running a 1Gbps fiber link

We have an identical setup actually working with a 10Gbps fiber link but we didnt think of trying that.

We'll try both the Copper link and the 10Gbps fiber link and if necessary a downgrade to 16.12.04 which was working previously to see if it helps.

Have you been able to find a way to make it work with 1gpbs fiber link on version 17.X ?

Thank you

[helpdeskdan]

We've noticed a problem between Cisco & certain other vendors where the port mysteriously starts having high packet loss.  We assured it was neither the optic itself nor the fiber by testing different devices and optics.  You can literally recreate this bug and get the switch port in a bad state by unplugging and re-plugging the optic a few times with a fiber run to certain non-Cisco devices.  (In this case, a Nokia 7705 sar-18) You can not recreate the problem if said fiber is to a 2960.  (And don't think I didn't try six ways to Sunday in hopes of getting it so)

There's only two ways I found to fix an port in a bad (packet loss) state.  The first is to plug the fiber that goes into the bad port, temporarily, into another Cisco device (I had a small 2960 to test) and ping it.  Somehow, this straightens the port out every time.  The second is to simply re-seat the optic.  Both cause the issue to disappear, but I found no way to fix it remotely.  No amount of shutting the port, re-seating fiber, or anything else besides possibly a reboot will cause the port to function correctly.  (The alarming part is that an upgrade caused the issue for us)

I've no time to open a tac case and, with it happening to "non Cisco" devices, I can't imagine said case going anywhere.

[balaji.bandi]

Always running stable gold version is advisable.

try no speed negotiation (that should fix some issue)

can you post show interface output - is this SFP cisco ? what mode cables ?