FTDv100 in Azure: Packet loss to internet

[ronnie.shih]

Hello, we have 4 standalone FTDv100 units in Azure running in production.  Once in a while, a specific unit would experience packet loss to the internet, anywhere from 10 to even 30% packet loss sometimes.  I basically do 1000 pings in sys support diag mode to 8.8.8.8 with default timeout of 2 seconds, so I'm not swamping google's public IP getting rate limited.  Rebooting the NVA in Azure does not make a difference.  I've opened tickets with both Cisco TAC and Azure support, basically the 2 vendors point fingers at each other siting "it's your NVA" from Azure support or "it's your virtualization environment" from Cisco TAC.  Obviously, as a customer, I cannot check on the Azure host hosting the FTDv vm.  This is what I used to do on vmware hosts and I would find physical network cards or physical disks getting overloaded, no such option in Azure.

Looking for guidance on how to troubleshoot this and find out the cause.  I will redeploy the FTDv onto a different host this weekend to move it onto a different host to see if problem persists.

[MHM Cisco World]

Can you capture traffic Outbound and match the number you cap with number of ping.

MHM

[ronnie.shih]

I did do this already with the last Cisco TAC ticket.  What TAC said is "hey look, we did send 1000 packet but got 880 back, so NIC is not having a delay with overloaded buffer, so some traffic coming in from the internet circuit is getting dropped" so the finger points back to Azure's virutualization environment as the fault.

[MHM Cisco World]

You ping 100 (no need 1000 for easy to count loss) check the icmp request not icmp reply' 

This give hint that indeed ftdv os not issue but the azure is issue.

MHM

[ronnie.shih]

capture cap1 interface outside match icmp any host 8.8.8.8 echo

capture cap2 interface outside match icmp any host 8.8.8.8 echo-rep

then I ping 100 times

I see 100 packets in cap1 and I see 85 packets in cap2 

This is the same thing I did with Cisco last time.  Then at this point they said it's your Azure virtualization environment.  The FTD only receives what the gateway can send back to it.  This is 15% loss.

[MHM Cisco World]

Do same but ping from any point behind the FTD not directly from FTD, there is limit rate of icmp in ftd it can effect ping and drop some if it exceed this limit

MHM

[ronnie.shih]

Already done that, that's where the first indication of packet loss is detected, major complaints from end users vpn-ed into the FTD.  Then I found out it's more obvious from the outside interface to the internet.  It is not a rate limit thing, not by 1 bit.  I have 3 functioning FTDs with only 1 to 5% packet loss, not this unit with 10% packet loss to the internet.

[ronnie.shih]

Packet loss issue resolved, by redeploying the FTDv vm which moves the vm onto a different Azure vm host.

This completely raises my suspicion that some azure hosts are getting overloaded by whatever activities that go on, on that specific host, and end customers have no way of identifying.

[MHM Cisco World]

I check issue if I found something useful I will send it to you PM.

Thanks for update me 

MHM