HTTP access outbound and back ...

[mfsumption]

I have a search engine spider that runs on my server that is protected by a PIX 501 with a basic configuration. The spider needs HTTP outbound access and back inbound again to spider a web site that is on the same server. I am a newbie, but I believe that somehow the firewall is blocking the inbound (re-entry so to speak) of the spider and therefore the spider is giving me errors that it cannot find the web site. Any ideas on how I could verify this and/or make a setting to allow this in a specific secure manner. Thanks.

[mfsumption]

No fix yet unfortunately. If the http traffic is open both ways then why can't the spider connect? I had the exact same problem with another utility called WebKeepAlive, which was installed on the same server, and all it does is make an http request to the sites on that same server, only I was getting a similar message stating that it couldn't connect. So there's got to be something blocking, and I guess I have to setup logging per Vibhor's suggestion. I don't think logging is turned on, so I have to figure out how to turn it on and try to simulate the spider activity to log something. You mentioned in your prior post that the PIX can't do "uturns" but I'm not exactly sure what you mean. The spider and the WebKeepAlive programs are just making an http request from the server to a URL that resolves to a web site on that very same server. That's the issue in the nutshell. Thanks for any additional suggestions.

[acomiskey]

Exactly...

Ok, for instance, lets say the inside web server is 192.168.1.1. You access this (abc.com) from the outside with something like...

static (inside,outside) 209.1.1.1 192.168.1.1 netmask 255.255.255.255

access-list outside_in permit tcp any host 209.1.1.1 eq 80

Let say the website is abc.com. This resolves to to it's public address 209.1.1.1. Does the spider server resolve abc.com to 209.1.1.1 or it's inside address 192.168.1.1?

My point is that if it resolves to 209.1.1.1, this will not work as it is currently configured. If it resolves to 192.168.1.1, then all should work fine. What does the spider server use for DNS? If you put yourself on the same network as the spider server and use the same dns, can you access abc.com?

[vitripat]

Hi all .. here is what I believe the way spider server is working.

Spider does a lookup for a website http://abc.com, now this website is actually hosted on the same (spider) server .. is this correct ?

In this scenario, the nslookup will result in the public IP of the spider server itself, and then spider server will try to access itself using the public IP ?? If this is the case, then this will not work due to U-Turning, which is not allowed on PIX-501.

However, you can try using DNS doctoring. I'm assuming that the internal IP address of the spider server is 10.0.0.2. In this case, try using following commands-

no static (inside,outside) [ip addr #2] 10.0.0.2

static (inside,outside) [ip addr #2] 10.0.0.2 dns

clear xlate

Now do "ipconfig /flushdns" on the spider server and all the internal hosts also.

Check now if things work.

Regards,

Vibhor.

[fred.s.mollenkopf · Accepted solution]

Michael,

There are a couple things that can prevent this.

1. PIX routing. Unless running 7.x and even then only with configuration changes to the default, the PIX doesn't allow routing back out an interface it received the inbound packet on. So if the web client(WebKeepAlive) on your web server is essentially making an http request to itself, it'll resolve DNS(assuming your using public) and receive it's Public IP. It will then route it's packet to it's default gateway (unless you have it specified in your web server route table) and that will probably be the PIX. The PIX will receive this and will eventually drop it due to security not allowing routing back out its source interface.

The easiest way to get around this for your scenario is to update the HOSTS file on the server with the Web Site FQDN using the Private IP and not the Public. DNS will never get invoked because the HOSTS file will resolve first. You will never hit the PIX and will be able to Spider your website for your reports or whatever.

I'm not going to discuss the other things that could block it because I'm pretty sure you ain't running 7.x on a 501 because it isn't supported. If it was 7.x you could loop the connection and then the thread could go on and on with Static commands and access-lists. Though you could technically use the DNS fixup on the static when it makes the DNS request but I would have to look that up. You could also configure routing on your web server for the Public IP but the HOSTs file is your best bet.

Please rate any helpful posts

Thanks

Fred

[acomiskey]

Correct, or do dns rewrite/doctoring. Looks like I was right yesterday after all....

[mfsumption]

Fred, thank you for joining this "conversation." The HOSTS file solution appeared to be the simplest as compared to the "DNS doctoring" so I tried it, and the good news is ... it worked! Thanks again!

Michael

[mfsumption]

Fred, Vibhor, and acomiskey: I must say that this is the finest, most prompt, and most helpful forum I have ever visited. I am a software developer now, though I used to be a network guy a long time ago, so I really appreciate all your help, and I will be happy to vote you all "high" 5's. Thanks again!