PIX 515 No traffic on new IP block

[dsc_tech_1]

We have been assigned a new range of ips 213.x.x.x/28 from our ISP. They are being routed via our existing gateway 92.x.x.146.

The problem:
We can't get any traffic to this pix on the new range 213.x.x.x/28.
- If we try to ping 213.x.x.61 we get Time to live exceeded.
- ISP gets the same from their router.
- ISP tries ssh and gets No route to host.

The ISP has checked and double checked the routing and the MAC address of our outside interface. They are correct.

The strange thing is we can't see ANY log messages relating to the new range for inbound connection attempts. The Pix is running at log level 7.

Does anyone have any idea what the problem might be? or any suggestions for debugging the issue?

Config extract:
Standalone Pix 515 running 7.0(7)
outside 92.x.x.146 255.255.255.240
inside 192.168.101.1 255.255.255.0
global (outside) 1 interface
nat (inside) 1 0.0.0.0 0.0.0.0
route outside 0.0.0.0 0.0.0.0 92.x.x.145 1
access-group acl_out in interface outside
access-list acl_out extended permit tcp any host 213.x.x.x eq www
access-list acl_out extended permit tcp any host 213.x.x.x eq ssh
static (inside,outside) 213.x.x.61 192.168.101.99 netmask 255.255.255.255
icmp permit any unreachable

192.168.101.99 is a linux test server with http and ssh

Any help much appreciated.

PM

[Jon Marshall]

dsc_tech_1 wrote:

We have been assigned a new range of ips 213.x.x.x/28 from our ISP. They are being routed via our existing gateway 92.x.x.146.
The problem:
We can't get any traffic to this pix on the new range 213.x.x.x/28. 
- If we try to ping 213.x.x.61 we get Time to live exceeded. 
- ISP gets the same from their router.
- ISP tries ssh and gets No route to host.
The ISP has checked and double checked the routing and the MAC address of our outside interface. They are correct.
The strange thing is we can't see ANY log messages relating to the new range for inbound connection attempts. The Pix is running at log level 7.
Does anyone have any idea what the problem might be? or any suggestions for debugging the issue?
Config extract:
Standalone Pix 515 running 7.0(7)
outside 92.x.x.146 255.255.255.240
inside 192.168.101.1 255.255.255.0
global (outside) 1 interface
nat (inside) 1 0.0.0.0 0.0.0.0
route outside 0.0.0.0 0.0.0.0 92.x.x.145 1
access-group acl_out in interface outside
access-list acl_out extended permit tcp any host 213.x.x.x eq www 
access-list acl_out extended permit tcp any host 213.x.x.x eq ssh
static (inside,outside) 213.x.x.61 192.168.101.99 netmask 255.255.255.255
icmp permit any unreachable
192.168.101.99 is a linux test server with http and ssh
Any help much appreciated.
PM

As long as you haven't disabled sysopt proxy-arp on the outside interface then it should work fine, and if it's not then it really does sound like an ISP issue. Could you post full config and remove any sensitive information.

Jon

[dsc_tech_1]

Hi Jon

Here is the full config minus the names section. Ethernet2 is not used and has no connectivity.

PIX Version 7.0(7)
!
hostname pix
domain-name y.com
enable password xxx encrypted
no names
various names...
dns-guard
!
interface Ethernet0
nameif outside
security-level 0
ip address 92.x.x.146 255.255.255.240
!
interface Ethernet1
nameif inside
security-level 100
ip address 192.168.101.1 255.255.255.0
!
interface Ethernet2
nameif net2
security-level 50
ip address 10.0.1.1 255.255.255.0
!
passwd xxx encrypted
boot system flash:/image
ftp mode passive
same-security-traffic permit inter-interface
access-list acl_out extended permit tcp 213.x.x.128 255.255.255.192 host 92.x.x.154 eq ssh
access-list acl_out extended permit icmp any any
access-list acl_out extended permit tcp any host 92.x.x.150 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.151 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.155 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.153 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.155 eq www
access-list acl_out extended permit tcp any host 92.x.x.156 eq www
access-list acl_out extended permit tcp any host 92.x.x.156 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.152 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.152 eq www
access-list acl_out extended permit tcp any host 92.x.x.157 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.158 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.158 eq www
access-list acl_out extended permit tcp any host 92.x.x.158 eq https
access-list acl_out extended permit tcp any host 92.x.x.158 eq smtp
access-list acl_out extended permit tcp any host 92.x.x.158 eq pop3
access-list acl_out extended permit tcp any host 92.x.x.158 eq imap4
access-list acl_out extended permit tcp any host 92.x.x.158 eq 10025
access-list acl_out extended permit tcp any host 92.x.x.158 eq 1863
access-list acl_out extended permit tcp any host 92.x.x.158 range 25000 30000
access-list acl_out extended permit tcp 213.x.x.128 255.255.255.192 host 92.x.x.15 eq ssh
access-list acl_out extended permit tcp any host 92.x.x.158 eq 8080
access-list acl_out extended permit tcp any host 92.x.x.158 eq 1020
access-list acl_out extended permit tcp any host 92.x.x.150 eq domain
access-list acl_out extended permit udp any host 92.x.x.150 eq domain
access-list acl_out extended permit udp any host 92.x.x.150 eq dnsix
access-list acl_out extended permit tcp any host 92.x.x.151 eq domain
access-list acl_out extended permit udp any host 92.x.x.151 eq domain
access-list acl_out extended permit udp any host 92.x.x.151 eq dnsix
access-list acl_out extended permit tcp any host 213.x.x.1 eq ssh
access-list acl_out extended permit tcp any host 213.x.x.1 eq www
access-list acl_in extended permit tcp any any
access-list acl_in extended permit udp any any
access-list acl_in extended permit icmp any any
access-list acl_cap extended permit ip any host 213.x.x.1
access-list acl_cap extended permit tcp any host 213.x.x.1
access-list acl_cap extended permit icmp any host 213.x.x.1
access-list acl_cap extended permit udp any host 213.x.x.1
no pager
logging enable
logging standby
logging console errors
logging monitor notifications
logging buffered debugging
logging trap debugging
logging history warnings
logging recipient-address x@y.com level critical
logging facility 22
logging host inside 192.168.101.10
logging debug-trace
mtu outside 1500
mtu inside 1500
mtu net2 1500
ip verify reverse-path interface outside
ip verify reverse-path interface inside
ip verify reverse-path interface net2
icmp permit any inside
icmp permit any unreachable inside
asdm history enable
arp timeout 14400
nat-control
global (outside) 1 interface
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) 92.x.x.150 192.168.101.10 netmask 255.255.255.255
static (inside,outside) 92.x.x.153 192.168.101.30 netmask 255.255.255.255
static (inside,outside) 92.x.x.155 192.168.101.31 netmask 255.255.255.255
static (inside,outside) 92.x.x.156 192.168.101.32 netmask 255.255.255.255
static (inside,outside) 92.x.x.152 192.168.101.33 netmask 255.255.255.255
static (inside,outside) 92.x.x.157 192.168.101.34 netmask 255.255.255.255
static (inside,outside) 92.x.x.158 192.168.101.21 netmask 255.255.255.255
static (inside,outside) 92.x.x.154 192.168.101.40 netmask 255.255.255.255
static (inside,outside) 92.x.x.151 192.168.101.11 netmask 255.255.255.255
static (inside,outside) 213.x.x.1 192.168.101.99 netmask 255.255.255.255
access-group acl_out in interface outside
access-group acl_in in interface inside
established tcp 0 0 permitto tcp 113 permitfrom tcp 0
route outside 0.0.0.0 0.0.0.0 92.x.x.145 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00
timeout mgcp-pat 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server RADIUS protocol radius
no snmp-server location
no snmp-server contact
snmp-server community xxx
snmp-server enable traps snmp authentication linkup linkdown coldstart
telnet 192.168.101.0 255.255.255.0 inside
telnet timeout 60
ssh timeout 5
console timeout 0
dhcpd dns 208.67.222.222
dhcpd lease 3000
dhcpd ping_timeout 750
dhcpd domain y.com
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map global_policy
class inspection_default
  inspect dns maximum-length 512
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
  inspect ils
!
service-policy global_policy global
tftp-server inside 192.168.101.10 pix-latest-config

Cryptochecksum:xxx

[Jon Marshall]

Can't see anything obviously wrong withj your config. What happens if you try to traceroute to it from the internet ? I would do this myself but obviously don't know the full IP

Jon

[dsc_tech_1]

traceroute to 213.x.x.1 (213.x.x.1), 64 hops max, 40 byte packets
1  10.0.1.1  1.974 ms  1.497 ms  1.529 ms
2  * * *
3  * * *
4  * * *
5  195.x.x.223  14.416 ms  13.726 ms  19.239 ms
6  217.x.x.82  14.348 ms  14.707 ms  14.444 ms
7  217.x.x.81  13.747 ms  13.852 ms  13.792 ms
8  217.x.x.82  15.212 ms  15.430 ms  15.425 ms
9  217.x.x.81  14.963 ms  18.489 ms  22.029 ms
10  * 217.x.x.82  42.106 ms  45.332 ms
11  217.x.x.81  42.501 ms  44.113 ms  43.964 ms
12  217.x.x.82  47.384 ms  20.220 ms  22.275 ms
13  217.x.x.81  21.453 ms  26.393 ms  18.243 ms
14  * 217.x.x.82  51.615 ms  50.746 ms
15  217.x.x.81  45.397 ms  30.960 ms  22.761 ms
16  217.x.x.82  29.489 ms  56.783 ms  43.424 ms
17  217.x.x.81  44.986 ms  43.618 ms  41.921 ms
18  217.x.x.82  47.819 ms *  34.886 ms

I get the same loop with 217.x.x.81, 217.x.x.82 when I use an online traceroute tool.

[Jon Marshall]

dsc_tech_1 wrote:
traceroute to 213.x.x.1 (213.x.x.1), 64 hops max, 40 byte packets
 1  10.0.1.1  1.974 ms  1.497 ms  1.529 ms
 2  * * *
 3  * * *
 4  * * *
 5  195.x.x.223  14.416 ms  13.726 ms  19.239 ms
 6  217.x.x.82  14.348 ms  14.707 ms  14.444 ms
 7  217.x.x.81  13.747 ms  13.852 ms  13.792 ms
 8  217.x.x.82  15.212 ms  15.430 ms  15.425 ms
 9  217.x.x.81  14.963 ms  18.489 ms  22.029 ms
10  * 217.x.x.82  42.106 ms  45.332 ms
11  217.x.x.81  42.501 ms  44.113 ms  43.964 ms
12  217.x.x.82  47.384 ms  20.220 ms  22.275 ms
13  217.x.x.81  21.453 ms  26.393 ms  18.243 ms
14  * 217.x.x.82  51.615 ms  50.746 ms
15  217.x.x.81  45.397 ms  30.960 ms  22.761 ms
16  217.x.x.82  29.489 ms  56.783 ms  43.424 ms
17  217.x.x.81  44.986 ms  43.618 ms  41.921 ms
18  217.x.x.82  47.819 ms *  34.886 ms
I get the same loop with 217.x.x.81, 217.x.x.82 when I use an online traceroute tool.

So what are the devices 217.x.x.81 and 217.x.x.82 ?

Clearly the packet is not getting to your but we need to know who owns the above devices and what they are. Suspect they may be routers owned by your ISP.

Jon

[Kureli Sankar]

Here is what you need to do.

1. Get the MAC address of your outside interface (sh int e0) .

2. Call your ISP Give that MAC address to your ISP and ask them if they see that listed as the MAC address to hand packets destined to 213.x.x.x/28 on their router.

-KS

[dsc_tech_1]

I have spoken to the ISP and confirmed the MAC address of the outside interface Ethernet0

ISP says
...we are sending this correctly to your pix, you should see any traffic destined for a 213.x.x.0/28 address hit your interface at 92.x.x.146/32

Yes 217.x.x.81 and 217.x.x.82 are routers owned by our ISP.

Is there anything else I can ask the ISP in terms of testing/debugging? I've run out of ideas.

[Jon Marshall · Accepted solution]

dsc_tech_1 wrote:
I have spoken to the ISP and confirmed the MAC address of the outside interface Ethernet0
ISP says
...we are sending this correctly to your pix, you should see any traffic destined for a 213.x.x.0/28 address hit your interface at 92.x.x.146/32
Yes 217.x.x.81 and 217.x.x.82 are routers owned by our ISP.
Is there anything else I can ask the ISP in terms of testing/debugging? I've run out of ideas.

If the routers are owned by your ISP then the fault lies with them. They have a routing loop in their network and that is why the packets are not getting to your firewall. Have you shown them the traceroute ?

They need to look at the .81 and .82 routers to work out why packets are looping between these 2 routers. Until they fix this packets will never get to your firewall.

Jon

[dsc_tech_1]

This issue has now been resolved.

I showed the ISP the traceroute and asked them to confirm everyting is 100% correct on their routers for this block.

The ISP found a configuration problem with thier bgp communities. Apparently we were the first client to request an additional IP block on this router.

Jon, thank you very much for you help.