Cant reach a service on a public IP...

rudepeople · ‎08-27-2019

We have a /28 block of public IP addresses pointed at one of our colo locations. We're using nat to pass port specific traffic through to individual servers behind the router which is an ISR4321k9.

There were a few bumbs but we have things working for the most part, however; the servers behind the router are not able to see each other via their public IPs.

Without going into too much detail, we have two servers who talk to each other through gitlab. the trouble is they do this using hostnames which are bound to those public IPs. But when they attempt to communicate with each other, they both get connection refused.

I have a VPN connection to the colo and I'm seeing the same activity over the vpn, when I try to go to the webpage of one of the servers while connected to the vpn, I get connection refused. but when I disconnect, the page comes up.

Please note, this isn't a DNS issue, when I ping the hostname while on the vpn, the correct IP comes up and replies. we just get a connection refused. I believe its a feature of the router blocking traffic from bouncing out to the internet for intranet resources, but I'm not sure how to address it.

In case it's useful, here's my config:

Router1#sh run
Building configuration...


Current configuration : 4185 bytes
!
! Last configuration change at 14:21:49 UTC Tue Aug 27 2019
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname slc-colo-net1
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
enable secret 5 [OMMITED]
enable password [OMMITED]
!
no aaa new-model
!
transport-map type persistent webui https-webui
 secure-server
!
transport-map type persistent webui http-webui
 server
!
transport-map type persistent webui http-https-webui
 server
 secure-server
!
!
!
!
!
!
!
!
!
!
!
!


ip name-server x.x.x.222

!
ip dhcp pool 1
 utilization mark high 80 log
 utilization mark low 70 log
 network 192.168.60.0 255.255.255.0
 domain-name int.pos.ac
 default-router 192.168.60.1
 dns-server x.x.x.222 8.8.8.8
 lease 30
!
!
!
!
!
!
!
!
!
!
subscriber templating
multilink bundle-name authenticated
!
!
!
crypto pki trustpoint TP-self-signed-1403732793
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1403732793
 revocation-check none
 rsakeypair TP-self-signed-1403732793
!
!
crypto pki certificate chain TP-self-signed-1403732793
license udi pid ISR4331/K9 sn [OMMITED]
!
spanning-tree extend system-id
!
username [OMMITED] privilege 15 secret 5 [OMMITED]
!
redundancy
 mode none
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface GigabitEthernet0/0/0
 description WAN000
 ip address x.x.x.40 255.255.255.240 secondary
 ip address x.x.x.37 255.255.255.240 secondary
 ip address x.x.x.38 255.255.255.240 secondary
 ip address x.x.x.39 255.255.255.240 secondary
 ip address x.x.x.41 255.255.255.240 secondary
 ip address x.x.x.42 255.255.255.240 secondary
 ip address x.x.x.43 255.255.255.240 secondary
 ip address x.x.x.44 255.255.255.240 secondary
 ip address x.x.x.45 255.255.255.240 secondary
 ip address x.x.x.46 255.255.255.240 secondary
 ip address x.x.x.36 255.255.255.240
 ip nat outside
 negotiation auto
!
interface GigabitEthernet0/0/1
 description WAN001
 ip address x.x.x.37 255.255.255.240
 ip nat inside
 shutdown
 negotiation auto
!
interface GigabitEthernet0/0/2
 no ip address
 shutdown
 negotiation auto
!
interface GigabitEthernet0/1/0
!
interface GigabitEthernet0/1/1
!
interface GigabitEthernet0/1/2
!
interface GigabitEthernet0/1/3
!
interface GigabitEthernet0/1/4
!
interface GigabitEthernet0/1/5
!
interface GigabitEthernet0/1/6
!
interface GigabitEthernet0/1/7
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 negotiation auto
!
interface Vlan1
 ip address 192.168.60.1 255.255.255.0
 ip nat inside
!
ip nat pool NAT1 x.x.x.36 x.x.x.36 netmask 255.255.255.240
ip nat inside source static tcp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat inside source static udp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat inside source static tcp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat inside source static udp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat inside source static tcp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat inside source static udp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat inside source static tcp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat inside source static udp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat inside source static tcp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat inside source static tcp 192.168.60.11 10000 interface GigabitEthernet0/0/0 10000
ip nat inside source static udp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat inside source list 1 interface GigabitEthernet0/0/0 overload
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 x.x.x.33
!
!
access-list 1 permit 192.168.60.0 0.0.0.255
!
snmp-server community public RO
!
!
control-plane
!
!
line con 0
 password [OMMITED]
 login
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 password [OMMITED]
 login
!
transport type persistent webui input https-webui
!
!
end

Richard Burts · ‎08-27-2019

You mention a vpn. But I do not see anything about vpn in the config that you posted. Can you tell us more about the vpn? Where is it running?

HTH

Rick

HTH

Rick

Giuseppe Larosa · ‎08-27-2019

Hello rudepeople,

your understanding is correct, the two servers should use their private IP addresses to make a successful communication.

To achieve this you can modify the local hosts file on each server adding an entry for the other one pointing to the private IP address so that DNS is not used for this entry.

with the extendable keyword the router is able to accept connections started from the outside mapping them to the right private IP address and port, but this does not work for communication between two servers that are actually both inside.

They should use their private IP addresses to communicate directly.

Hope to help

Giuseppe

paul driver · ‎08-28-2019

Hello

Another alternative would be to change the NAT to be domainless ( no inside/outside) this way the NAT order is changed so the router performs routing decisions before and after translation which this case should allow your internal host reach an internal server via it natted public ip address.

Example:
ip nat source static tcp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat source static udp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat source static tcp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat source static udp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat source static tcp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat source static udp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat source static tcp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat source static udp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat source static tcp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat source static tcp 192.168.60.11 10000 interface GigabitEthernet0/0/0 10000
ip nat source static udp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat source list 1 interface GigabitEthernet0/0/0 overload

int gig0/0/1
no ip nat inside
ip nat enable

int gig0/0/0
no ip nat outside
ip nat enable

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

rudepeople · ‎08-28-2019

@Richard Burts wrote:
You mention a vpn. But I do not see anything about vpn in the config that you posted. Can you tell us more about the vpn? Where is it running?

client vpn, using openVPN. nothing within the router itself.

@Giuseppe Larosa wrote:
Hello rudepeople,
your understanding is correct, the two servers should use their private IP addresses to make a successful communication.

Awe... I was hoping that wouldn't be the case. the way the webdevs have built the site, the software references the public IPs a ton and changing it now would be costly.

Looks like I'm going to have to nat the public IPs directly to the servers... good thing they both have dual NICs and adequate firewalls.

@paul driver wrote:
Hello
Another alternative would be to change the NAT to be domainless ( no inside/outside) this way the NAT order is changed so the router performs routing decisions before and after translation which this case should allow your internal host reach an internal server via it natted public ip address.

Example:
ip nat source static tcp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat source static udp 192.168.60.21 22 x.x.x.40 22 extendable
ip nat source static tcp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat source static udp 192.168.60.21 80 x.x.x.40 80 extendable
ip nat source static tcp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat source static udp 192.168.60.21 443 x.x.x.40 443 extendable
ip nat source static tcp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat source static udp 192.168.60.21 9558 x.x.x.40 9558 extendable
ip nat source static tcp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat source static tcp 192.168.60.11 10000 interface GigabitEthernet0/0/0 10000
ip nat source static udp 192.168.60.11 1194 interface GigabitEthernet0/0/0 1194
ip nat source list 1 interface GigabitEthernet0/0/0 overload

int gig0/0/1
no ip nat inside
ip nat enable

int gig0/0/0
no ip nat outside
ip nat enable

this seems promising... however you no ip nat outside set on gi0/0/1... why? that port is shut and I have no intention of actually using it... the servers are connected through int vlan 1 (physically using the NIM ports gi0/1/0 - gi0/1/7), if that makes a difference.

[EDIT]: @paul driver - never mind... I just realized you had that there because I had the port configured with an IP and route. that's old. We were going to try to set a redundant wan but we only have one subnet so it was pointless. I just haven't removed it yet.

Giuseppe Larosa · ‎08-28-2019

Hello ,

>> Awe... I was hoping that wouldn't be the case. the way the webdevs have built the site, the software references the public IPs a ton and changing it now would be costly.

I am not a professional software developer, I think that most of SW developers rely on hostnames so the suggestion to create two entries in hosts file looked like reasonable.

By the way, finding and replacing the public IP address with the private IP address should not be so difficult unless the number of source code files is very high.

It would be better to have a text file out of source code where all these parameters are set and then code can read the configuration file to inizialize variables.

I did so in the past with PERL or TCL/TK scripts.

Hope to help

Giuseppe