Problem in accessing the http and https services using ASA very strange

[imranbhatti151]

Hi All,

I need help from security gurus.

I am unable to access the web sites and below error message appears when opening.

Some times websites are opened but most of time very slow and timed out. Extremely slow access.

92.122.208.146|80|172.19.110.21|1419|Teardown TCP connection 1045559 for outside:92.122.208.146/80 to inside:172.19.110.21/1419 duration 0:00:30 bytes 0 SYN Timeout

172.19.110.21|1275|172.19.107.1|443|Deny TCP (no connection) from 172.19.110.21/1275 to 172.19.107.1/443 flags FIN ACK  on interface inside

I am fedup these error , can some one help me

I have below system versions

ASA   8.0 ( 2)

ASDM 6.2

[Jennifer Halim]

Base on the error message, SYN timeout, it means that the server is not responding to TCP 3 way handshake as it is timing out on the TCP SYN packet.

You might want to check if the server is up and running correctly, and check the interface speed and duplex on the server. Is there any other server within the same zone that has the issue? or this is the only server that is having issue? If it's the only server who's having the issue, it's more likely a server issue than the ASA issue.

[imranbhatti151]

I also  think that 3 way TCP handshake is not going to complete

But this is happening all the hosts ( inside to outside means) accessing the internet from inside to hosts.

I am unable to access  the internet most of the time , some time web sites appears and normally this error comes.

can this be due to the slow internet access ?

i have outside interface connected to VSAT internet modem and i am getting the 750 ~900 ms delay when trying to ping to yahoo.com

Also worth mentioning that FTP protocol has no issue .

could it be due to VSAT intenet or due to misconfiguration of ASA

Please help

Thanks

[shzaman]

Hi,

As mentioned in previous reply first message is about SYN timeout meaning the SYN went out but till 30 seconds we didn't receive any SYN+ACK, easy way to verify this or have a close look is to put captures on inside and outside interfaces and see the packet flow, if we see SYN going out and nothing is coming back then there is something causing problem after ASA (on outside) so it can be because of packet drop (may be congestion/slow link) but there can be other possibilities also depending upon network setup/environment.

Second message means that we received packet with FIN+ACK but the connection doesn't exist on firewall meaning firewall doesn't have that in connection table.

Here are few suggestions/points in addition to the reply from Jennifer, check packet captures to verify the behavior, if you bypass ASA (connect host directly to internet) then you still observe these issues? (if yes then I think we have found the problem ) . Make sure that in network topology host s supposed to send traffic to ASA and ASA is sending that out to ISP (meaning no Asymmetric routing or issues at lower layer causing packets). For testing you may try bypassing ASA and one other test if needed is to connect host directly to ASA.

As far as FTP is concerned that also involves three-way handshake so if FTP is used frequently then you should face problem with that one also unless traffic for FTP is not following same path (FTP on outside, client on inside) or the server we are trying for web has issue or it is not occurring much.

I hope this helps.

-Shahid

[imranbhatti151]

Dear Shahid,

Thank you very much for reply and support.

I understand

My Firewall outside interface is directly connected with the Modem of the VSAT. No other things( device)  in between.

If i directly connect host to the modem , Internet working perfectly.

So whats wrong , i changed the cable between firewall and VSAT modem , nothing works

Only option i left is to consider some problem in VSAT modem device . But VSAT provider asking that direct pc have no problem so his device is functioning properly.

Please advise

Regards

Imran

[shzaman]

Hi,

I am not sure about inside topology, have you tried to connect inside PC directly to ASA and test that. And share information about inside topology also. Check 'show asp drop' command and see if you are seeing any drops there , if yes then take that output and save it for reference then use 'clear asp drop' command to clear those counters and try to browse and after some time check if you are seeing any drops 'show asp drop' and share that output.

Also share some information about device configuration, is there anything special configured like changed MSS or timeout values or traffic policing and what about HTTP inspection, is it enabled (try disabling that if possible). Also check CPU/memory usage also, is that showing anything abnormal.

I think you will be interested in this video also, starting from 6th minute(approx)  Kureli Sankar (Cisco Engineer) is discussing about the latency issues through firewall.

Link: https://supportforums.cisco.com/videos/1075

I hope this will help.

-Shahid 

[shzaman]

Wanted to add one point, first check 'show int' output for speed/duplex settings and if there are any errors (CRC, collision etc.), if yes try clearing those and check after some time. By the way the link that I mentioned will have more information.

[imranbhatti151]

Dear Shahid

Thank you very much for reply

Yes Drops has been seen on firewall

You can see out put below also no abnormal things like memory and CPU has been observed

Frame drop:
  No valid adjacency                                        552
  No route to host                                          273
  Flow is denied by configured rule                     2502848
  First TCP packet not SYN                                24127
  TCP data send after FIN                                     6
  TCP failed 3 way handshake                                692
  TCP RST/FIN out of order                                 2805
  TCP packet SEQ past window                                 57
  TCP invalid ACK                                             8
  TCP Out-of-0rder packet buffer full                        87
  TCP Out-of-Order packet buffer timeout                     35
  TCP RST/SYN in window                                    1918
  TCP DUP and has been ACKed                            3169118
  Slowpath security checks failed                          3713
  IP option drop                                           1396
  ICMP Error Inspect different embedded conn                 12
  DNS Inspect invalid packet                                 17
  DNS Inspect invalid domain label                          104
  DNS Inspect packet too long                                12
  DNS Inspect id not matched                                 13
  Interface is down                                          45
  Dropped pending packets in a closed socket               3434

Flow drop:
  NAT failed                                                 14
  Inspection failure                                       2002
  SSL bad record detected                                   217
  SSL received close alert                                    6

Also i have seen below config in firewall

tcp-map mss-map
  exceed-mss allow

Below are configurations for inspect

class-map inspection_default
match default-inspection-traffic
class-map http-map1
match access-list http-list2
!
!
policy-map global_policy
class inspection_default
  inspect rsh
  inspect rtsp
  inspect sqlnet
  inspect skinny
  inspect sunrpc
  inspect xdmcp
  inspect sip
  inspect netbios
  inspect tftp
policy-map type inspect dns migrated_dns_map_1
parameters
  message-length maximum 512
policy-map http-map1
class http-map1
  set connection advanced-options mss-map
!
service-policy global_policy global

Please advise

[Oscar Cardiel]

Hi there.

has your server two NICs?

[shzaman]

Hi,

Clear the drops using command 'clear asp drop' then after some time check how these are increasing and you may capture the packets getting dropped in ASP by using 'capture cap1 type asp-drop all' command and then do 'sh cap cap1' to check packets and use 'sh cap cap1 | in ' to see packets dropped for specific IP.

Information about these drop reasons can be found on link: http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/s2.html#wp1351326

Capture command help: http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c1.html#wp2108895

What about interface stats and as mentioned by user: Oscar about NIC cards? and have you tested by connecting PC directly to ASA? Also share output of 'sh service-policy' and 'sh asp drop' after clearing those and allow some time to pass.

-Shahid

[imranbhatti151]

HI Shahid

Thanks again for supporting me.

Below is the output after clearing and clearly showing that it is increasing

Interface stats are normal no crc and collissions and we have only one NIC in proxy server and also in other pcs

Sh asp drop

Frame drop:
  Flow is denied by configured rule                       77026
  First TCP packet not SYN                                  540
  TCP failed 3 way handshake                                 14
  TCP RST/FIN out of order                                   39
  TCP DUP and has been ACKed                                148
  Slowpath security checks failed                           222
  Dropped pending packets in a closed socket                  2

Flow drop:
  Inspection failure                                         90

Capture result is  ( note inside interface  is a 802.1q interface with two vlan( subinterfaces) , 107 and 301) 172.19.108.15 is our proxy server

F01# sh cap cap1 | in 172.19.108.15

  91: 10:38:10.315688 802.1Q vlan#107 P0 172.19.108.15.58153 > 209.85.146.147.80
: R 3801422935:3801422935(0) win 0
109: 10:38:11.545442 802.1Q vlan#107 P0 172.19.108.15.59455 > 4.23.54.126.80: F
2671847652:2671847652(0) ack 1948589870 win 65535
110: 10:38:11.545442 802.1Q vlan#107 P0 172.19.108.15.59580 > 4.23.54.126.80: F
158076312:158076312(0) ack 1566687617 win 65535
111: 10:38:11.545442 802.1Q vlan#107 P0 172.19.108.15.52683 > 4.23.54.126.80: F
296836186:296836186(0) ack 1325884648 win 65535
124: 10:38:13.025206 802.1Q vlan#107 P0 172.19.108.15.62868 > 217.163.21.37.80:
R 947079666:947079666(0) win 0
838: 10:39:08.774923 802.1Q vlan#107 P0 172.19.108.15.65386 > 172.19.109.11.25:
R 2405507004:2405507004(0) win 0
911: 10:39:14.903792 802.1Q vlan#107 P0 172.19.108.15.52887 > 164.46.230.36.80:
F 11072808:11072808(0) ack 303114077 win 65535
912: 10:39:14.903792 802.1Q vlan#107 P0 172.19.108.15.50048 > 164.46.230.36.80:
F 376466281:376466281(0) ack 1223761515 win 65535
1094: 10:39:50.212986 802.1Q vlan#107 P0 172.19.108.15.62417 > 209.85.146.139.80
: F 212350602:212350602(0) ack 1911057388 win 65535
1095: 10:39:50.212986 802.1Q vlan#107 P0 172.19.108.15.63620 > 209.85.146.113.80
: F 499560277:499560277(0) ack 2065700036 win 65535
1096: 10:39:50.212986 802.1Q vlan#107 P0 172.19.108.15.64893 > 218.145.28.57.80:
F 2683057573:2683057573(0) ack 2098978771 win 65535
2552: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.57191 > 124.83.230.247.80
: F 2402672053:2402672053(0) ack 408318225 win 65535
2553: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.59067 > 157.166.255.22.80
: F 1173855759:1173855759(0) ack 467500409 win 65535
2554: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.57435 > 218.145.28.57.80:
F 3958388812:3958388812(0) ack 188205860 win 65535
2555: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.61206 > 218.145.28.227.80
: F 3292236323:3292236323(0) ack 832932232 win 65535
2556: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.50084 > 93.184.220.33.80:
F 2001697565:2001697565(0) ack 1150791319 win 65535
2557: 10:41:20.496403 802.1Q vlan#107 P0 172.19.108.15.51883 > 59.106.108.72.80:
F 3044666264:3044666264(0) ack 927250299 win 65535
3091: 10:41:45.611784 802.1Q vlan#107 P0 172.19.108.15.54656 > 209.85.146.155.80
: R 2312296347:2312296347(0) win 0
3271: 10:41:54.180227 802.1Q vlan#107 P0 172.19.108.15.49255 > 209.85.146.19.80:
R 477269530:477269530(0) win 0
3787: 10:42:57.896560 802.1Q vlan#107 P0 172.19.108.15.53249 > 172.19.109.11.25:
R 3667559968:3667559968(0) win 0
4073: 10:43:28.800892 802.1Q vlan#107 P0 172.19.108.15.56241 > 209.85.146.104.80
: R 2897547054:2897547054(0) win 0
4541: 10:43:50.796849 802.1Q vlan#107 P0 172.19.108.15.49429 > 209.85.146.99.80:
R 3330175187:3330175187(0) win 0
4548: 10:43:50.896819 802.1Q vlan#107 P0 172.19.108.15.60681 > 209.85.146.99.80:
R 2132491876:2132491876(0) win 0
5404: 10:44:31.055127 802.1Q vlan#107 P0 172.19.108.15.49199 > 209.85.146.99.80:
R 1050183837:1050183837(0) win 0
5684: 10:44:44.452643 209.85.146.113.80 > 172.19.108.15.55546: . 568089835:56809
1215(1380) ack 1131064961 win 32240

Service policy output is

Global policy:
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: rsh, packet 0, drop 0, reset-drop 0
      Inspect: rtsp, packet 0, drop 0, reset-drop 0
      Inspect: sqlnet, packet 38395, drop 0, reset-drop 0
      Inspect: skinny , packet 0, drop 0, reset-drop 0
      Inspect: sunrpc, packet 0, drop 0, reset-drop 0
      Inspect: xdmcp, packet 0, drop 0, reset-drop 0
      Inspect: sip , packet 139, drop 0, reset-drop 0
      Inspect: netbios, packet 29459, drop 0, reset-drop 0
      Inspect: tftp, packet 0, drop 0, reset-drop 0

I have not tried directly connecting PC to inside of firewall as then i have to remove the 802.1Q setting from inside interface.

Please advice

Thanks

[shzaman]

Hi,

I think these drops are for approx. 1 hrs (just guessing after looking at posts time), you may find detail about these drops on link mentioned in my previous message. I am not sure about complete flow of traffic like complete packet exchange (we may go for checking that) but I can see reset from proxy server and it is for port:80 traffic mainly, here is a sample

3271: 10:41:54.180227 802.1Q vlan#107 P0 172.19.108.15.49255 > 209.85.146.19.80:
R 477269530:477269530(0) win 0
3787: 10:42:57.896560 802.1Q vlan#107 P0 172.19.108.15.53249 > 172.19.109.11.25:
R 3667559968:3667559968(0) win 0
4073: 10:43:28.800892 802.1Q vlan#107 P0 172.19.108.15.56241 > 209.85.146.104.80
: R 2897547054:2897547054(0) win 0
4541: 10:43:50.796849 802.1Q vlan#107 P0 172.19.108.15.49429 > 209.85.146.99.80:
R 3330175187:3330175187(0) win 0
4548: 10:43:50.896819 802.1Q vlan#107 P0 172.19.108.15.60681 > 209.85.146.99.80:
R 2132491876:2132491876(0) win 0

In previous posts you mentioned that FTP works fine so FTP is also passing through the proxy or not? Here are few suggestion which should take us to conclusion

--Try to connect PC directly to ASA and see the performance (whenever it is possible) or may be if you have any interface free on ASA then configure that to have access to internet and connect system on that one.

--You may try bypassing the proxy also.

--One more point that on switch side on trunk interface only allow Vlans which are configured on ASA.

--Also share output of 'sh run all sysopt'

To me it looks like the issue is with internal side but we have to confirm that.

I hope this will help you.

-Shahid

[imranbhatti151]

Thanks Again,

Bypassing proxy also gives same result

I am actually very surprised that some time even we got very good speed and all services looks like resumed but after few minutes , situation becomes same.

on 301 vlan we also have same result.

Ftp is using direct connection without proxy but without proxy i am also unable to browse .

Below is the output of   sh run all sysopt

no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
no sysopt nodnsalias inbound
no sysopt nodnsalias outbound
no sysopt radius ignore-secret
sysopt connection permit-vpn
no sysopt connection reclassify-vpn

I have checked On switch only two vlans are allowed  ( 107 and 301) on that trunk

interface GigabitEthernet1/0/12
description To ASA5510
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 107,301
switchport mode trunk
srr-queue bandwidth share 10 10 60 20
srr-queue bandwidth shape  10  0  0  0

--------------------------------------

Regards

Imran

[shzaman · Accepted solution]

Hi,

We may go for checking complete packet exchange while we have issue to see what is happening wrong or if possible you may open TAC ticket to have Webex with some Cisco engineer and check all necessary things (if possible like having support agreement etc). But in addition to my last replies here are few suggestions (some points are may be duplicate)

--Check switch side also for interface errors, speed/duplex because when you connect host directly to modem you observe normal speed so the things which are not there in that case are local LAN and ASA.

--Try connecting PC directly to ASA and check performance (already mentioned)

--Is there any other device doing layer-3 routing in network (other then ASA) then make sure your default gateways (for devices) are setup correctly and no asymmetric routing or redirection is happening.

--If you are not using any VPN (IPSec, SSL, GRE, l2tp etc.) then you may increase TCP MSS from 1380 to 1460 by using command 'sysopt connection tcpmss 1460' , this will allow more data to be there in TCP segment so it should effect performance positively and it depends upon applications also. But do this if you don't have any VPN.

--Check the video which I mentioned in my previous replies.

I hope this will help you.

-Shahid

[imranbhatti151]

Thank you very much all for support

Problem resolved and it was probably by due to performance issues on ISP side

Any way thank you very much specialy by Mr. Shahid.