Latency over IPSEC

[AirSail]

Hello Folks, 

running into a crazy issue here, I have a firewall cluster in Taiwan (remote)---> IPSEC VPN ---> Firewall cluster in UsA(HQ),  

the main usage of that VPN is to have all agents roughly 500 users access a Citrix portal, 

80% of the day things are normal, average latency 220ms which is normal due to geo distance/endcap decap...etc. 

but at some point, the latency goes up to 500ms for about a  few min, which results from a production drop as everyone needs to re-login, 

spent weeks on that issue, I did a lot of optimization, and got TAC engaged, but I still from time to time we run into latency issue, 

any one with the same story? any recommendation? what to capture if the issue happens? 

[Rob Ingram]

@AirSail what is the Firewall cluster, ASA or FTD?

What version of software are the firewalls running? - perhaps a bug for your specific version?

When the latency occurs, does the firewall hardware CPU/Memory spike?

Are you monitoring the firewall via an NMS so you can see if there is any issues around the time there is latency?

[Sheraz.Salim]

Have you log ticket to your WAN/Internet provider? worth checking if latency is from there end during the day. Do you have Netflow configured in your network?

Does this happen specific time day? when this issue occurs you need to take the statistics of the router/firewall. you have not mentioned what appliances you running. Need more information.

[AirSail]

I had many tickets with the ISP, as the issue is random, they just don't say anything useful, or they ask most of the time for trace.   for the NetFlow nop, I don't have one, but I'm thinking of it. 

good question about the timing, I would say yes, quite repetitive when all the users log in at the same time and start pulling applications from Citrix portal, - I worked with TAC they said the firewall hits 60% at peak time and that won't harm at all, the hardware is FPR2130, 

[Sheraz.Salim]

Have look at this document either you run FTD code or ASA code it shall put you in some right direction Here 

some of the command you can take the output are "show conn", "show process cpu non-zero" and show threat-detection statistics host", "show asp drops"

[MHM Cisco World]

For recommendations try use PMTU in ipsec vpn, it can the path in ISP core is change and hence mtu is change. 

Try use PMTU or reduce mtu to low value

MHM

[AirSail]

Hello @MHM Cisco World  - I checked show crypto ipsec sa and I see MTU 15000 that seems to be set to default, 

what is PMTU? what is that "P"   - any recommended MTU size to put that can't affect webapplication such as Citrix ? 

[MHM Cisco World]

PMTU is path MTU' it way FW use to detect mtu in path 

For mtu value use 1400.

MHM

[AirSail]

Sounds good I'll put an MTU of 1400, 

and for PMTU, is there something specific to configure for it? 

[MHM Cisco World]

You can enable it from vpn topolgy

Advanced > IPsec > IPsec Settings

Enable Fragmentation Before Encryption

This option lets traffic travel across NAT devices that do not support IP fragmentation. It does not impede the operation of NAT devices that do support IP fragmentation.

Path Maximum Transmission Unit Aging

Check to enable PMTU (Path Maximum Transmission Unit) Aging, the interval to Reset PMTU of an SA (Security Association)

Value Reset Interval

Enter the number of minutes at which the PMTU value of an SA (Security Association) is reset to its original value. Valid range is 10 to 30 minutes, default is unlimited.

MHM

[AirSail]

Hello MHM and team, 

I'm still spending the day looking at different parameters and logs, and I started capturing, and I captured many logs such as: 

[https 443] drop rate-1 exceeded. current burst rate is 8 per second max ....

and other logs showing [port range] or [scanning] .... 

i did further research and it sounds this has something to do with Thread-detection, 

looking at some commands "show threat-detection statistics host | b myCitrixserver" the numbers are huge  comparing to other entries showing on that long output

Thread-detection could affect/drop traffic going to Citrixserver? 

is there any specific command to put an exception for a specific IP to not be proceeded by thread detection? 

thanks a lot !

[AirSail]

Anyone with further updates ? 

[AirSail]

@HMHMHM @Rob Ingram @Sheraz.Salim  - any feedback regarding the latesting reply I posted, do you think Thread-detection engine could lead to similar behavior ?