Observing Intermediate packet drops...

shantilal · ‎06-23-2022

We are observing intermediate packet drops in traffic where the cisco C9300-24T is connected to fortigate firewall.

We are using copper port on switch side and sfp-RJ45 at fortigate side.

Need help to resolve this issue. Thanks.

Georg Pauwen · ‎06-23-2022

Hello,

can you post the output of:

show interfaces x

where 'x' is the interface connected to the Fortigate ?

shantilal · ‎06-23-2022

Hi,

We are actually migrating the firewall. The previous firewall had copper port and there were no any packet loss observed.

But the new firewall has all the SFP ports & we are using SFP-RJ45 module for copper connectivity between switch & firewall.

And now we are facing intermediate packet drops in the traffic.

Georg Pauwen · ‎06-24-2022

Hello,

how do you notice the packet loss ? If there are no drops on the interface(s), do you have a management tool that shows you there is packet loss, or users complaining ?

Leo Laohoo · ‎06-24-2022

@shantilal wrote:

But the new firewall has all the SFP ports & we are using SFP-RJ45 module for copper connectivity between switch & firewall.

What happens if the connection goes straight to the switch copper ports (bypass/not-use the uplink modules)?

What firmware is the switch on?

shantilal · ‎06-24-2022

Switch is directly connected to firewall.

Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.6.9

Leo Laohoo · ‎06-24-2022

@shantilal wrote:

Switch is directly connected to firewall.

That's not the answer to my question: You said that the firewall is connected to the switch using a GLC-T module. What happens if the firewall is connected to the switch using the switch's copper ports (and not the SFP ports)?

shantilal · ‎06-24-2022

We are facing issue with the new firewall that has SFP ports.

We have used ping for finding drops. The source was behind the switch and destination was behind the firewall.

Source -----> Switch---->Firewall with SFP-RJ45 ------> Destination

Georg Pauwen · ‎06-24-2022

Hello,

the reason I am asking if the users are actually experiencing problems is that the Fortigate itself might cause the PING drops. According to Fortigate support, the below applies:

"This is an expected behavior: The package is dropped since the ICMP is exceeding the rate limit. The FortiGate team has a limitation for ICMP; the limit is 6 packets per second per sender. This is based on RFC 1812: 4.3.2.8 Rate Limiting A router which sends ICMP Source Quench messages MUST be able to limit the rate at which the messages can be generated. A router SHOULD also be able to limit the rate at which it sends other sorts of ICMP error messages (Destination Unreachable, Redirect, Time Exceeded, Parameter Problem). The rate limit parameters SHOULD be settable as part of the configuration of the router. How the limits are applied (e.g., per router or per interface) is left to the implementor's discretion."

shantilal · ‎06-24-2022

Hi Georg,

We have testing the ICMP traffic by bypassing the switch and directly connected two laptops to the firewall but we didn't observed any packet drops.

But If we connect the switch then we observed packet drops. Do we need to check rate limit on the switch side ?

Georg Pauwen · ‎06-24-2022

Hello,

since you do not see any drops on the interfaces of the switch (show interfaces x), you could use Wireshark or SPAN on the switch to analyze the traffic.

Can you post the output of:

show buffers

from the switch ?

shantilal · ‎11-23-2022

Hi Georg,

After a long time, I was checking the issue from the fortigate side but not found any issue yet.

Can you please any command or solution from the switch side. So, I can verify that switch is dropping the packets or not.

Should we use SPAN or any other methods are there ?

We are doing a UAT testing in which we create a separate VLAN in the switch & will connect a laptop & the fortigate for testing the traffic whether any drop observed or not.

Your support will be very helpful for us in the troubleshooting.

Thanks.

MHM Cisco World · ‎11-24-2022

you can share the show interface in SW
and interface in FW ??

shantilal · ‎11-24-2022

BFL_DC_NEW_DMZ_SW#sh int GigabitEthernet1/0/21
GigabitEthernet1/0/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 00b1.e305.1315 (bia 00b1.e305.1315)
Description: NEW_DMZ_SW to FGT_PRI_FW_PORT-05
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 56/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2917106843
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 10911000 bits/sec, 11154 packets/sec
5 minute output rate 220997000 bits/sec, 20246 packets/sec
148540391668 packets input, 73889634753821 bytes, 0 no buffer
Received 9639152 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
224097852349 packets output, 272420000481180 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

--------------------------Server Interface-------------------------------------------

BFL_DC_NEW_DMZ_SW#sh int Te1/1/5
TenGigabitEthernet1/1/5 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 00b1.e305.1321 (bia 00b1.e305.1321)
Description: ***CONNECTED TO SYNERGY CHASIS C3 RACK ***
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 1/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5437000 bits/sec, 1325 packets/sec
5 minute output rate 3362000 bits/sec, 1224 packets/sec
30534086522 packets input, 33369783361525 bytes, 0 no buffer
Received 191225066 broadcasts (182101967 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 182101967 multicast, 0 pause input
0 input packets with dribble condition detected
19767003742 packets output, 8833876173747 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

-------------------------------------After connect on 800C -----------------------------------
Fortinet Interface

BFL_DC_NEW_DMZ_SW#sh int status GigabitEthernet1/0/21
GigabitEthernet1/0/21 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 00b1.e305.1315 (bia 00b1.e305.1315)
Description: NEW_DMZ_SW to FGT_PRI_FW_PORT-05
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 58/255, rxload 10/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4140229225
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 40852000 bits/sec, 13843 packets/sec
5 minute output rate 230442000 bits/sec, 21977 packets/sec
148606239729 packets input, 73904977728714 bytes, 0 no buffer
Received 9640096 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input

------------------------------Server interface 172.30.1.127-----------------------------------

BFL_DC_NEW_DMZ_SW#sh int Te1/1/5
TenGigabitEthernet1/1/5 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 00b1.e305.1321 (bia 00b1.e305.1321)
Description: ***CONNECTED TO SYNERGY CHASIS C3 RACK ***
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 1/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5437000 bits/sec, 1325 packets/sec
5 minute output rate 3362000 bits/sec, 1224 packets/sec
30534086522 packets input, 33369783361525 bytes, 0 no buffer
Received 191225066 broadcasts (182101967 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 182101967 multicast, 0 pause input
0 input packets with dribble condition detected
19767003742 packets output, 8833876173747 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out