PIX 515E blocking internal-to-internal traffic

brian975 · ‎08-17-2004

I have a PIX 515E 6.3(1) that has started blocking traffic from certain inside machines to other inside machines.

I didn't think the PIX could filter from one interface to the same interface.

This is only happening on traffic across my routers that connect our two subnets.

None of this was happening until a power outage and UPS failure brought down our satellite office. A reboot of the servers and a flush of the ARP table on the switches has helped a little bit but we are still having lots of trouble.

----------------

Router1 is configured like this:

ES#show ip route

Gateway of last resort is 192.55.9.10 to network 0.0.0.0

C 192.55.9.0/24 is directly connected, Ethernet0

192.55.10.0/30 is subnetted, 1 subnets

C 192.55.10.0 is directly connected, Serial0

R 192.55.11.0/24 [120/1] via 192.55.10.2, 00:00:22, Serial0

S* 0.0.0.0/0 [1/0] via 192.55.9.10

----------------

Router2 is configured like this:

FH#show ip route

Gateway of last resort is 192.55.9.10 to network 0.0.0.0

R 192.55.9.0/24 [120/1] via 192.55.10.1, 00:00:16, Serial0

192.55.10.0/30 is subnetted, 1 subnets

C 192.55.10.0 is directly connected, Serial0

C 192.55.11.0/24 is directly connected, Ethernet0

S* 0.0.0.0/0 [1/0] via 192.55.9.10

----------------

And this is part of the PIX config:

name 207.243.xx.xxx NISC_access

access-list acl_out permit tcp any host 12.45.xx.xxx eq https

access-list acl_out permit tcp host NISC_access host 12.45.xx.xxx eq ssh log

access-list acl_out permit tcp any host 12.45.xx.xxx eq 3389

access-list acl_out permit tcp any host 12.45.xx.xxx eq https

access-list acl_out permit tcp any host 12.45.xx.xxx eq smtp

access-list acl_out permit tcp any host 12.45.xx.xxx eq www

access-list acl_out permit icmp any any echo-reply

access-list acl_out permit icmp any any unreachable

access-list acl_out permit icmp any any time-exceeded

access-list acl_out permit tcp host NISC_access eq telnet host 12.45.xx.xxx log

access-list inside_outbound_nat0_acl permit ip any 192.55.9.0 255.255.255.192

ip address outside 12.45.xx.xxx 255.255.255.0

ip address inside 192.55.9.10 255.255.255.0

global (outside) 10 interface

nat (inside) 0 access-list inside_outbound_nat0_acl

nat (inside) 10 0.0.0.0 0.0.0.0 0 0

static (inside,outside) tcp 12.45.xx.xxx 3389 192.55.9.5 3389 netmask 255.255.255.255 0 0

static (inside,outside) 12.45.xx.xxx 192.55.9.1 netmask 255.255.255.255 0 0

static (inside,outside) 12.45.xx.xxx 192.55.9.3 netmask 255.255.255.255 0 0

static (inside,outside) 12.45.xx.xxx 192.55.9.2 netmask 255.255.255.255 0 0

access-group acl_out in interface outside

rip inside passive version 2

route outside 0.0.0.0 0.0.0.0 12.45.36.1 1

----------------

This is the error that shows up in the PIX log:

<163>Aug 17 2004 12:00:42: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/135 dst inside:192.55.11.2/4430

<163>Aug 17 2004 12:00:48: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/135 dst inside:192.55.11.2/4430

<163>Aug 17 2004 12:01:00: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/135 dst inside:192.55.11.2/4439

<163>Aug 17 2004 12:01:03: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/135 dst inside:192.55.11.2/4439

<163>Aug 17 2004 12:01:09: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/135 dst inside:192.55.11.2/4439

<163>Aug 17 2004 12:01:19: %PIX-3-106011: Deny inbound (No xlate) tcp src inside:192.55.9.5/42 dst inside:192.55.11.2/2577

<163>Aug 17 2004 14:47:21: %PIX-3-106011: Deny inbound (No xlate) icmp src inside:192.55.9.1 dst inside:192.55.11.63 (type 8, code 0)

<163>Aug 17 2004 14:48:21: %PIX-3-106011: Deny inbound (No xlate) icmp src inside:192.55.9.1 dst inside:192.55.11.63 (type 8, code 0)

----------------

Mostly this is communication between domain controllers and users connecting to a mainframe.

Any help would be appreciated.

a.awan · ‎08-18-2004

What is the default gateway set to on the hosts on the 192.55.9.0/24 segment? Can you post a print out of the show route on the PIX firewall?

brian975 · ‎08-18-2004

The default gw on the 192.55.9.0 segment is our router on that segment: 192.55.9.210.

Here's the route on the PIX:

PIX515# show route

outside 0.0.0.0 0.0.0.0 12.45.xx.x 1 OTHER static

outside 12.45.xx.x 255.255.255.0 12.45.xx.xxx 1 CONNECT static

inside 192.55.9.0 255.255.255.0 192.55.9.10 1 CONNECT static

inside 192.55.10.0 255.255.255.252 192.55.9.210 1 RIP

inside 192.55.11.0 255.255.255.0 192.55.9.210 2 RIP

mostiguy · ‎08-18-2004

A "sysopt noproxyarp inside" should disable proxy arp on the inside int. The only reason in your config I can think of for the pix to be receiving that traffic is if it is giving out arp data that tells clients it should receive those packets.

Do you have any arp tables for afflicted clients?

a.awan · ‎08-18-2004

The PIX should not be seeing the traffic between your 192.55.99.0/x and the 192.168.11.0/x hosts. I think the best thing to do will be to take one machine having connectivity issues and try to trace what is going on when it tries to communicate to your remote site. The PIX has routes to the remote sites so my initial thought of proxy arp happening should not be the issue in this case.

brian975 · ‎08-18-2004

I was under such pressure to get things back up that I didn't really spend any time trying to diagnose what went wrong.

I flushed the arp cache on every network device in the org and rebooted just about everything. For the few machines that were still having trouble, a reboot solved the problem.

My mission now is to try to prevent this from happening again. That satellite office has never gone down all at once and, now that the inverter is fixed, it shouldn't but, I never want this issue to happen again.

Any thoughts would help. Should I disable proxy ARP?

a.awan · ‎08-18-2004

This is what i think might have transpired but no guarantees that it did happen this way :-).

From your description i have understood that ES is a router at the primary site and FH is the router at the remote site that lost power ... is this correct?

You lost power at your remote site and consequently the route to 192.55.11.0 got flushed from the routing table of the router at the primary site (the ES router). All hosts trying to reach 192.55.11.0 had their default gateway configured as 192.55.11.210 but when the packet reached this router it did not have a route to 192.55.11.0 and ended up sending it out using its default route to the PIX 192.55.9.10. It is at this point that i think the trouble started. When a router has to send a packet out the same interface that it receives it on because another device on the same network has a better route to it then it sends out an icmp redirect to the host originating the packet to basically tell it to use the better device for subsequent communication to that destination. I think at this point your hosts started using the PIX as the next hop for any devices that they wanted to talk to on the 192.55.11.0 subnet. Nothing malfunctioned but it is just the way your network is configured that it created a problem. Again this is my analysis of the situation and i would love to hear more comments on it from others.

How to avoid it in future, well i think there are two things that can be done and none of them involves disabling proxy arp. One is to disable ip redirects on the ethernet interface of the router connecting to 192.55.10.0 subnet. The second is to move the PIX to a separate segment provided you have the resources to do so. I hope all this makes sense.