So I'm running into an odd scenario with one of the software developers I'm working with, where one of his internal tools has recently started to have an increasing amount of TCP re-transmissions within our network configuration. Basically there's a large data packet followed immediately by a 1 byte closing packet, the contents of the data packet are showing up clean, but in some scenarios the 1 byte closing packet is forced to be re transmitted.
Through working with him we've diagnosed that the issue only occurs with data packets with specific byte lengths and appears to skip every 2 digits.
EX: Packets of 845, 846, 849, 850, 863, 864... bytes cause an issue with the closing 1 byte forced to be re-transmitted (adding 300ms delay per fault). However, packets sized 845, 846, 849, 850, 863, 854... don't have the issue with a failed re-transmission of the following 1 byte closing packet.
I've setup packet traces on both sides of the issue and have confirmed the pattern as well, along with setting up packet traces on our ASA's to rule out any potential ACL or routing issues on that end. We're seeing the TCP re-transmit issue on both the ingress and egress interfaces on the ASA device, so I'm comfortable ruling out any ACL or routing issues on the ASA end.
However, when relocating the client (VLAN_X) onto the same switch as the server (VLAN_Y) with IP routing enabled, the single dropped 1 byte packet issue disappears for all data packet sizes. So the issue logically appears to be that this 1 byte packet is being lost/dropped somewhere between transmission from the switch to the directly connected ASA, but only with specific packet sizes.
The environment simplified has a single 2960CX setup in IP routing mode with 3 VLANs (X, Y, Z). X can host clients (as seen in the relocation note above), Y hosts the server, and Z is the VLAN reserved for the ASA interconnect.
I'm curious if anyone's had a similar issue like this with specific packet sizes and orders resulting in consistantly re transmitted packets.
Working on building a stripped down configuration file of the switch to upload, and will update in a bit.
