Hi, I am troubleshooting an issue where we see intermittant packet loss within our infrastructure. The setuip looks like this.
webserver = vlan120
webcache = vlan120
vlan120 is a dmz with fwsm as Layer3 gateway
fwsm svi is on vlan 119
ace module is frontend vlan 119 backend vlan 120
Layer3
webserver(1.1.1.1/24) --> webcache(1.1.1.2/24)
we will see comms errors on the servers causing a session to timeout, we experience approx 30 per night causing clients to recieve a 500 error.
I have tried pinging between the boxes using the following command but cannot reproduce the issue.
ping -c 10000 -s 1300 -i 0 1.1.1.1
10000 packets transmitted, 10000 received, 0% packet loss, time 12490ms
rtt min/avg/max/mdev = 0.908/1.179/11.445/0.718 ms, ipg/ewma 1.249/1.091 ms
Now if we do the same ping from the fwsm to the webserver with the following command we lose between 5 - 10 packets every 10000.
ping 1.1.1.1 size 1300 repeat 10000
Success rate is 99 percent (9994/10000), round-trip min/avg/max = 1/1/10 ms
This is the only way to reliably 'lose' packets. A tcpdump on the destination however reveals it receives 10000 icmp request packets and transmits 10000 icmp reply packets but the fwsm reports a loss.
Not sure what is going on here!
My questions are:
1) is the fwsm ping test valid?
2) where are my packets going?
3) how do i troubleshoot the (possibly 2) issues further
