Re: Failing to load some webpages (perhaps cgi)

foundationis · ‎12-21-2009

I am behind an ASA 5510 and there are 2 sites in particular that are giving me errors.

When I try to access: http://ssdi.genealogy.rootsweb.com/cgi-bin/ssdi.cgi

I get a "Problem loading page" error. "the connection to the server was reset" by firefox. The same error happens in IE.

Yet, if I go to another machine that doesn't go through the ASA 5510, I can access the page with no problems at all. We have another timeout issue that is experianced through an IPSec tunnel and we think that these two issues may go hand-in-hand with one another.

The ASA real time logging shows the connection happening, "192.168.2.90 Accessed URL 66.43.27.25:/cgi-bin/ssdi.cgi"

I have inspection setup as below:

!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
description ftp
class inspection_default
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect dns preset_dns_map
inspect ils
inspect ftp
inspect icmp
inspect icmp error
inspect http
!
service-policy global_policy global

any thoughts? Thanks for all help!

Kureli Sankar · ‎12-21-2009

What is the reason for inspect http? Company policy? If not pls. try to remove

conf t

policy-map global_policy
class inspection_default

no inspect http

Then give it a shot and let us know.

-KS

foundationis · ‎12-21-2009

the same error is still happening.

updated:

class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
description ftp
class inspection_default
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect dns preset_dns_map
inspect ils
inspect ftp
inspect icmp
inspect icmp error
!
service-policy global_policy global

EDIT**

Attached is the config. let me know if you see something out of the ordinary

Kureli Sankar · ‎12-21-2009

Interesting. What is the difference between your computer breaking and another computer working through this same firewall?

Do both these computers look like the same IP address when they go out to the internet? or diff. IP address.

Any other device like websense or other content scanning device monitoring the traffic doing some sort of policing?

Next step is to collect captures to see why the connection is getting reset and where the reset is coming from.

Let me know if you need instructions to configure captures. You can find many threads in the forum for that.

-KS

foundationis · ‎12-21-2009

I'll try to explain a bit better.

When I try to go through the ASA from a computer on the internal network, the site does not work. We've tested this out on several machines on the network.

If I configure a machine to NOT go out through the ASA, but bypass it using an external IP address, then the webpage displays properly. So there has to be a problem with the ASA.

Both routes have different external IP addresses assigned to them, but they are in the same block of addresses assigned from our ISP.

We do not have Websense or any filtering. There is a direct connection from the core switch to the inside interface of the ASA.

I will do some data capture if necessary. Can you give me some links? I'm familiar with wireshark, but only if I have it installed on my PC and not collecting information from around the whole entire network.

Kureli Sankar · ‎12-21-2009

Here is an action plan for you.

1. Take the public IP address that you used on the PC to test outside the firewall and use that as a global to translate this one test computer on the inside.

2. Clear local x.x.x.x where x.x.x.x is the address of the PC on the inside

3. On the inside PC, when you go to http://ipchicken.com it should show you the address that worked on the outside computer.

4. Test these websites that fail from the inside PC. If it still fails collect wireshark capture on the PC and upload it for us to look at.

5. If you are running 7.2.4 or above code (otherwise you need to use acl and apply capture) on the ASA you can try this simple capture lines on the asa. Here is the link that explains how to use access-list: http://security-planet.de/2005/07/26/cisco-pix-capturing-traffic/

cap capin int inside match tcp host x.x.x.x any eq 80

cap capout int outside match tcp host y.y.y.y any eq 80

where x.x.x.x is the inside address and y.y.y.y is the same address that worked when the PC was on the outside, that you are using now to translate the inside host on the inside.

sh cap capin

sh cap capout

clear cap capin

clear cap capout

-KS

foundationis · ‎12-22-2009

KS,

thanks for helping out so much.

As a result of me following steps 1-4, it works! If i set a static NAT for my PC using a different IP it works. but why? We can't use a different IP for the dynamic NAT because it would break an IPSec tunnel we are using to another ASA.

Here is what the log shows when I use your method in steps 1-4 using my own IP of 199.X.X.88:

6|Dec 22 2009|12:49:19|302014|66.43.27.37|80|192.168.3.54|4864|Teardown TCP connection 83470 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4864 duration 0:00:00 bytes 641 TCP FINs
6|Dec 22 2009|12:49:19|302013|192.168.3.54|4865|74.125.91.149|80|Built outbound TCP connection 83471 for 900SprintT1ISP:74.125.91.149/80 (74.125.91.149/80) to inside:192.168.3.54/4865 (199.X.X.88/4865)
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4864|66.43.27.37|80|Built outbound TCP connection 83470 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4864 (199.X.X.88/4864)
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4863|Teardown TCP connection 83468 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4863 duration 0:00:00 bytes 604 TCP FINs
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4862|Teardown TCP connection 83467 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4862 duration 0:00:00 bytes 602 TCP FINs
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4860|Teardown TCP connection 83465 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4860 duration 0:00:00 bytes 600 TCP FINs
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4861|Teardown TCP connection 83466 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4861 duration 0:00:00 bytes 599 TCP FINs
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4859|Teardown TCP connection 83464 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4859 duration 0:00:00 bytes 599 TCP FINs
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4858|Teardown TCP connection 83463 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4858 duration 0:00:00 bytes 938 TCP FINs
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4863|66.43.27.37|80|Built outbound TCP connection 83468 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4863 (199.X.X.88/4863)
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4862|66.43.27.37|80|Built outbound TCP connection 83467 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4862 (199.X.X.88/4862)
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4861|66.43.27.37|80|Built outbound TCP connection 83466 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4861 (199.X.X.88/4861)
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4860|66.43.27.37|80|Built outbound TCP connection 83465 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4860 (199.X.X.88/4860)
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4859|66.43.27.37|80|Built outbound TCP connection 83464 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4859 (199.X.X.88/4859)
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4857|Teardown TCP connection 83462 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4857 duration 0:00:00 bytes 938 TCP FINs
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4858|66.43.27.37|80|Built outbound TCP connection 83463 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4858 (199.X.X.88/4858)
6|Dec 22 2009|12:49:18|302014|66.43.27.37|80|192.168.3.54|4856|Teardown TCP connection 83460 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4856 duration 0:00:00 bytes 942 TCP FINs
6|Dec 22 2009|12:49:18|302013|192.168.3.54|4857|66.43.27.37|80|Built outbound TCP connection 83462 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4857 (199.X.X.88/4857)
6|Dec 22 2009|12:49:17|302014|66.43.27.37|80|192.168.3.54|4853|Teardown TCP connection 83454 for 900SprintT1ISP:66.43.27.37/80 to inside:192.168.3.54/4853 duration 0:00:00 bytes 937 TCP FINs
6|Dec 22 2009|12:49:17|302013|192.168.3.54|4856|66.43.27.37|80|Built outbound TCP connection 83460 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4856 (199.X.X.88/4856)
6|Dec 22 2009|12:49:17|302013|192.168.3.54|4855|66.43.27.37|80|Built outbound TCP connection 83459 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4855 (199.X.X.88/4855)
6|Dec 22 2009|12:49:17|302014|66.43.27.25|80|192.168.3.54|4852|Teardown TCP connection 83453 for 900SprintT1ISP:66.43.27.25/80 to inside:192.168.3.54/4852 duration 0:00:00 bytes 943 TCP FINs
6|Dec 22 2009|12:49:17|302013|192.168.3.54|4854|66.43.27.37|80|Built outbound TCP connection 83455 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4854 (199.X.X.88/4854)
6|Dec 22 2009|12:49:17|302013|192.168.3.54|4853|66.43.27.37|80|Built outbound TCP connection 83454 for 900SprintT1ISP:66.43.27.37/80 (66.43.27.37/80) to inside:192.168.3.54/4853 (199.X.X.88/4853)
6|Dec 22 2009|12:49:17|302013|192.168.3.54|4852|66.43.27.25|80|Built outbound TCP connection 83453 for 900SprintT1ISP:66.43.27.25/80 (66.43.27.25/80) to inside:192.168.3.54/4852 (199.X.X.88/4852)

Here is what the log looks like when I am using the Dynamic NAT which everyone else uses (ip 199.X.X.79):

6|Dec 22 2009|12:51:59|302014|64.233.169.148|80|192.168.3.54|4881|Teardown TCP connection 83738 for 900SprintT1ISP:64.233.169.148/80 to inside:192.168.3.54/4881 duration 0:01:02 bytes 4828 TCP Reset-I
6|Dec 22 2009|12:51:59|302014|74.125.91.149|80|192.168.3.54|4884|Teardown TCP connection 83741 for 900SprintT1ISP:74.125.91.149/80 to inside:192.168.3.54/4884 duration 0:01:02 bytes 43806 TCP Reset-I
6|Dec 22 2009|12:51:59|302014|205.247.221.57|80|192.168.3.54|4885|Teardown TCP connection 83742 for 900SprintT1ISP:205.247.221.57/80 to inside:192.168.3.54/4885 duration 0:01:02 bytes 1327 TCP Reset-I
6|Dec 22 2009|12:51:59|302014|209.90.101.200|80|192.168.3.54|4887|Teardown TCP connection 83744 for 900SprintT1ISP:209.90.101.200/80 to inside:192.168.3.54/4887 duration 0:01:01 bytes 985 TCP Reset-I
6|Dec 22 2009|12:51:59|302014|209.90.101.200|80|192.168.3.54|4886|Teardown TCP connection 83743 for 900SprintT1ISP:209.90.101.200/80 to inside:192.168.3.54/4886 duration 0:01:01 bytes 782 TCP Reset-I
6|Dec 22 2009|12:51:49|302014|66.43.27.25|80|192.168.3.54|4892|Teardown TCP connection 83815 for 900SprintT1ISP:66.43.27.25/80 to inside:192.168.3.54/4892 duration 0:00:04 bytes 952 TCP Reset-O
6|Dec 22 2009|12:51:44|302013|192.168.3.54|4892|66.43.27.25|80|Built outbound TCP connection 83815 for 900SprintT1ISP:66.43.27.25/80 (66.43.27.25/80) to inside:192.168.3.54/4892 (199.X.X.79/8932)
6|Dec 22 2009|12:51:44|305011|192.168.3.54|4892|199.X.X.79|8932|Built dynamic TCP translation from inside:192.168.3.54/4892 to 900SprintT1ISP:199.X.X.79/8932
6|Dec 22 2009|12:51:44|302014|66.43.27.25|80|192.168.3.54|4891|Teardown TCP connection 83812 for 900SprintT1ISP:66.43.27.25/80 to inside:192.168.3.54/4891 duration 0:00:04 bytes 684 TCP Reset-O
6|Dec 22 2009|12:51:39|302013|192.168.3.54|4891|66.43.27.25|80|Built outbound TCP connection 83812 for 900SprintT1ISP:66.43.27.25/80 (66.43.27.25/80) to inside:192.168.3.54/4891 (199.X.X.79/57683)
6|Dec 22 2009|12:51:39|305011|192.168.3.54|4891|199.X.X.79|57683|Built dynamic TCP translation from inside:192.168.3.54/4891 to 900SprintT1ISP:199.X.X.79/57683

Why is the connection getting Reset-O?

Kureli Sankar · ‎12-22-2009

So it works when you use a diff. IP. Glad to hear.

We have seen this problem many times. Reset-O means a reset came from the lower security interface.

There could be many reasons for why one IP in the same subnet works but the other doesn't.

Here they are.

1. ISP load balancing algorithm may sent one IP - in one way and the other IP via another way.

2. The website that you are hitting they may block one IP but allow the other one.

3. The website that you are trying to hit may do a reverse lookup on the IP that you look like and allow based on whether you have an "A" record created or not.

Find out all the diff. between these two IPs from the DNS, ISP and website admins and clear it up.

As far as the firewall is concerned, you don't have to change the dynamic NAT that is already there. Just for this one host you can use a separate nat/global pair and not bother the rest.

nat (inside) 100 192.168.3.54 255.255.255.255

global (outside) 100 w.w.w.w

where w.w.w.w is a working address.

or you can use policy nat as well. Only when you try to reach this website you can look like like w.w.w.w

-KS

foundationis · ‎12-22-2009

I've ping'ed the ISP and that website so now it looks like I have to wait for a response.

it just doesn't make sense to me why that would be happening. There are a few people that need to access this site. it would be a real pain to do Static Policy NAT rules for every website that ends up giving us trouble.

Thanks for the help!

Kureli Sankar · ‎12-22-2009

You don't have to change the outside IP of the firewall and disturb the tunnel.

You can just change everybody's global to this working IP address.

That should be easy to do.

Wait and see what the ISP and the website admins say.

I have given you all the reasons that I could think of. This is not a firewall problem. We can certainly prove that will captures.

-KS

pavlosd · ‎02-06-2010

Hi we are actually facing similar problems and I was wondering if you managed to find the exact problem that caused all this. The problem seems to be with some web pages "hosted" to various online service providers.

Also could you elaborate a bit more on the DNS issue that kusankar mentioned with the reverse lookup?

Kureli Sankar · ‎02-06-2010

I have added one more to the reasons below that I posted.

1. ISP load balancing algorithm may sent one IP - in one way and the other IP via another way.

2. The website that you are hitting they may block one IP but allow the other one.

3. The website that you are trying to hit may do a reverse lookup on the IP that you look like and allow based on whether you have an "A" record created or not.

4. MSS issues - where we have to allow them via MPF.

A discovery has been made that there are a few HTTP servers on the Internet that do not honor the MSS that the client advertises. Subsequently, the HTTP server sends data packets to the client that are larger than the advertised MSS. Before release 7.0, these packets were allowed through the PIX Security Appliance. With the security enhancement included in the 7.0 software release, these packets are dropped by default. This document is designed to assist the PIX/ASA Security Appliance administrator in the diagnosis of this problem and the implementation of a workaround to allow the packets that exceed the MSS.

You would most certainly see these syslogs

%ASA-4-419001: Dropping TCP packet from outside:192.168.9.2/80 to 
inside:192.168.9.30/1025, reason: MSS exceeded, MSS 460, data 1440

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a00804c8b9f.shtml

pixfirewall(config)#access-list http-list2 permit tcp any eq 80
pixfirewall(config)#class-map http-map1
pixfirewall(config-cmap)#match access-list http-list2    
pixfirewall(config-cmap)#exit
pixfirewall(config)#tcp-map mss-map
pixfirewall(config-tcp-map)#exceed-mss allow
pixfirewall(config-tcp-map)#exit
pixfirewall(config)#policy-map http-map1
pixfirewall(config-pmap)#class http-map1
pixfirewall(config-pmap-c)#set connection advanced-options mss-map
pixfirewall(config-pmap-c)#exit
pixfirewall(config-pmap)#exit
pixfirewall(config)#service-policy http-map1 interface outside

Outside interface is the internet facing interface. You can add this to the existing policy-map that is already applied
globally as well.

#3 above means that let us say when you go to the internet you look like 1.1.1.1 the destination is let us say google.com. Google will look
to see if 1.1.1.1 has a name associated with it like web.xyz.com or smtp.xyz.com or something.

-KS

pavlosd · ‎02-08-2010

Hi kusankar,

We partially managed to solve the problem. It seems that some Web Server Hosting Providers are blocking the IP addresses ending with .255 and .0, even if they are part of a bigger than a class C subnet... Maybe as a rule of thumb for some attacks?

But we are still having a problem with a specific service provider that has apache proxy in front and iis service on the backend. the servers respond to Error 404 - File not found when we access them from the PAT address, while if i do one-to-one nat the page works fine.

The TCP MSS Exceed Issue, I though it was "solved" or better say "disabled" with releases 8.0(x) http://www.ciscosystems.md/en/US/docs/security/asa/asa80/release/notes/arn804n.html

Kureli Sankar · ‎02-08-2010

Hmm interesting... I can understand denying .255 and .0 but how can the end server know whether you are hiding behind PAT or behind 1-1 STATIC?

May be it bases it off of the number of connections arrive from a particular source address.

-KS

pavlosdm · ‎02-08-2010

You are right though :-). What's more interesting is that the hosting service provider was somehow (apache module? or someking of inline filtering spam?) using DNSBL, blocking our PAT IP Address!

I accententaly found out that our PAT address is listed in DNS BL Lists (i.e. http://www.spamhaus.org/). I removed it from the list and after a while the web pages was working!!!

I asked our web server experts and our ISP for such a technique and while they were supriced as well.... This is mostly used only for mail servers and not for web...