TCP RESET DROPPED BY FIREWALL

[Magesh Kumar]

Hi Team,

We are currently encountering an unusual issue within our network.

According to the traces conducted at the servers, it appears that the TCP RESET packet sent by the Source Server is not being received by the Destination Server.

It is noteworthy that only the TCP RESET packets are being dropped in transit, while all other TCP packets are successfully passing through without any issues. Both the source and destination servers are situated in different locations and are connected via two Cisco ASA firewalls.

Could it be possible that the firewall is blocking TCP RESET packets? Even though there is no existing traffic flow in the connection table, should these packets not be permitted to pass through via a new match on the firewall?

Furthermore, considering that these servers are hosted on ESXi hypervisors, are there any specific settings that could result in the dropping of TCP RST packets?

Thanks !

Regards!

[M02@rt37]

Hello @Magesh Kumar 

ESXi hypervisor may also contribute to the issue. Its virtual switch settings, such as "Forged Transmits" or "Promiscuous Mode," could block packets if they are disabled. Try to adjust these settings to allow forged transmissions could resolve the problem. Additionally, NIC offloading features like TCP checksum offloading or MTU mismatches in the network path could cause certain packets, including TCP RESET, to be dropped. Disabling these offloading features or ensuring consistent MTU configurations can mitigate this risk.

Another potential issue is PMTUD. If the TCP RESET packets exceed the MTU size on the network path and ICMP "Fragmentation Needed" messages are blocked, the packets might be dropped. Verify packet fragmentation using packet captures and ensuring ICMP is not filtered will help diagnose and resolve this problem.

To pinpoint where the packets are being dropped, use packet captures on the ASA and the ESXi hypervisor. On the ASA, packet capture commands can help you determine if the packets are being received and forwarded. 

Cisco ASA firewalls are known to inspect and control TCP flows based on their connection table. If there is no matching entry for the flow, the firewall might drop out-of-context TCP RESET packets, assuming they are irrelevant or potentially malicious. To address this, you can verify the connection table using the show conn command and ensure that any TCP reset policies are correctly configured. Explicitly permitting TCP RESET packets in the ACL or enabling service resetinbound can help propagate the pakets through the Firewall.

[Magesh Kumar]

M02@rt37 Thanks for the detailed info.

You’re right. We’ve captured the traffic at the destination firewall (last L3 device on our setup), and it looks like the packets are getting there but then disappearing past the firewall, meaning at the ESXi hypervisors.

We’ve already checked the Promiscuous mode settings, and it's turned off.

Can you explain what other settings we should check on the hypervisors' side?

Thanks!

[M02@rt37]

You're so welcome @Magesh Kumar.

Since the TCP RESET packets are disappearing beyond the firewall and appear to be dropped at the ESXi hypervisors, several specific configurations should be checked. First, revisit the port group security policies on the virtual switch (vSwitch or Distributed Switch) where the affected virtual machines reside. Even though Promiscuous Mode is disabled, ensure that "Forged Transmits" is set to Accept. If it is set to Reject, the hypervisor might block packets with a source MAC address that doesn’t match the virtual machine’s MAC address. Similarly, verify that the "MAC Address Changes" setting is also set to Accept to allow legitimate MAC address adjustments by the VM's network stack.

Next, review the vNIC and driver configurations for the affected virtual machines. Confirm that the virtual NIC is using the VMXNET3 adapter type, which is optimized for performance and compatibility. Some legacy adapters, such as E1000, might struggle with specific traffic types like TCP RESET packets. Additionally, certain offloading features such as TCP Segmentation Offload and Large Receive Offload should be temporarily disabled to determine if they are interfering with packet processing. These features are typically enabled by default for performance but can sometimes cause unexpected behavior with certain packet types.

The MTU settings across the network path should also be consistent. If there is a mismatch, it could lead to fragmentation of TCP RESET packets, which the virtual switch or hypervisor might drop if not properly configured. Ensure that the MTU is uniform across the physical NICs, the virtual switch, and the VM network interfaces. Typically, this value is set to 1500 bytes, unless jumbo frames are explicitly configured and supported throughout the path.

[MHM Cisco World]

Can I see wireshark of traffic 

MHM