A strange packet loss in a switching environment.

[mrbob0473]

I have a switching environment with four switches as the diagram showing below.  SW1 and SW2 are two Cisco 6509-E switches, SW3 and SW4 are two Cisco 4848E switches, Host 1 and Host 2 are Solaris servers.

The original issue reported was that the SSH session between two servers were not be able to established.  Tcpdump result on the servers indicated there was packet loss in the network.  Instead of randomly lost some of the packets, only specific packets were lost in the network.

When running ssh server on Host2 and running ssh client on Host1, the Diff-Hellman Key Exchange Reply packet from Host2 to Host1 was lost.

When running ssh server on Host1 and running ssh client on Host2, the Diff-Hellman GEX Init pakcet from Host2 to Host1 was lost.

There are no ACL configured on the switch interfaces.

The packet loss seems targeted for Diff-Hellman related packets, which is really weird.  Does any one know why it happened?

[vietcisco]

I work with the individual reporting this issue and turns out a bad line card on the 6509-E where these 4948e access switches trunk to is the ultimate root cause.  I don't believe there were any syslog or other indicators of the failure and it was semi-masked by the fallout being dependent on where the VLAN roots were homed so it wasn't consistent.  Ultimately when the VLAN roots were reviewed in-depth, it did make complete sense on the impact being seen on just some access switches and ports and not others.

I believe I've written up all the pertinent facts here or perhaps left a few out and they will be filled in or corrected later.