Internal Hosts can ping internet but my routers cannot

[Chris.k88]

Hi,

So i have a issue with my home lab that i set up. I have 3 routers and 3 switches. The switches are all connected to my internal hosts and one of them is a L3 acting as the gateway for the hosts. The L3 switch is connected to one of the routers and the 3 routers all connected to each other with serial connections. The router the L3 connects to is acting as the NAT router that connects to my ISP home router (so double NAT going on)

All my switches and internal hosts can ping out to the internet but none of the 3 routers can, they can ping as far as my G0/0 interface on the router that is doing the NAT but not passed it. The G0/0 interface is getting it's IP via DHCP from the home ISP router.

Strange also is if i do a extended ping from my LAN interface of my router i can reach the internet but not from any other interface on any router.

This is my first time posting so please let me know what further information/configs you need or I am missing.

Thanks in advance!

[Meheretab Mengistu]

I am wondering whether you apply NAT for all internal IP addresses you have (including the serial interfaces IP addresses). Please provide the output of the following commands:-
On the the Router connected to the ISP : show run | s nat, show ip route, show run | s access-list
On one of the other routers or both routers: show run | s interface serial, show ip route

HTH,
Meheretab

[Chris.k88]

Hi Meheretab,

I have a permit any statement to not block any traffic, i read in some other forums that sometimes permit any statements cause problems with NAT(?) so at a point i had the specific statements to permit my subnets but it made no change so i put it back to permit any. Show commands are below.

Also after running traceroutes from the other routers i see the packets are getting to R1(Nat router) but seem to be dropping after that.

R1 (NAT router to ISP)

R1#show run | s nat
ip nat outside
ip nat enable
ip nat inside
default-information originate
ip nat inside source list NAT interface GigabitEthernet0/0 overload

S* 0.0.0.0/0 [254/0] via 192.168.0.1
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
C 10.10.0.0/30 is directly connected, Serial0/1/1
L 10.10.0.2/32 is directly connected, Serial0/1/1
O 10.10.0.4/30 [110/128] via 10.10.0.9, 00:01:11, Serial0/1/0
[110/128] via 10.10.0.1, 00:02:59, Serial0/1/1
C 10.10.0.8/30 is directly connected, Serial0/1/0
L 10.10.0.10/32 is directly connected, Serial0/1/0
172.16.0.0/16 is variably subnetted, 12 subnets, 2 masks
O 172.16.0.1/32 [110/65] via 10.10.0.9, 00:01:11, Serial0/1/0
O 172.16.1.1/32 [110/65] via 10.10.0.9, 00:01:11, Serial0/1/0
O 172.16.2.1/32 [110/65] via 10.10.0.9, 00:01:11, Serial0/1/0
O 172.16.3.1/32 [110/65] via 10.10.0.1, 00:02:59, Serial0/1/1
O 172.16.4.1/32 [110/65] via 10.10.0.1, 00:02:59, Serial0/1/1
O 172.16.5.1/32 [110/65] via 10.10.0.1, 00:02:59, Serial0/1/1
C 172.16.6.0/24 is directly connected, Loopback0
L 172.16.6.1/32 is directly connected, Loopback0
C 172.16.7.0/24 is directly connected, Loopback1
L 172.16.7.1/32 is directly connected, Loopback1
C 172.16.8.0/24 is directly connected, Loopback2
L 172.16.8.1/32 is directly connected, Loopback2
192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.0.0/24 is directly connected, GigabitEthernet0/0
L 192.168.0.108/32 is directly connected, GigabitEthernet0/0
192.168.3.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.3.0/24 is directly connected, GigabitEthernet0/1
L 192.168.3.1/32 is directly connected, GigabitEthernet0/1
O 192.168.4.0/24 [110/2] via 192.168.3.2, 00:01:21, GigabitEthernet0/1
O 192.168.20.0/24 [110/2] via 192.168.3.2, 00:01:21, GigabitEthernet0/1
O 192.168.30.0/24 [110/2] via 192.168.3.2, 00:01:21, GigabitEthernet0/1

R1#show run | s access-list
ip access-list standard NAT
permit any

R2

R2#show int s0/1/0
Serial0/1/0 is up, line protocol is up
Hardware is GT96K Serial
Internet address is 10.10.0.5/30
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
130 packets input, 10569 bytes, 0 no buffer
Received 54 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
119 packets output, 10889 bytes, 0 underruns
0 output errors, 0 collisions, 7 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

R2#show int s0/1/1
Serial0/1/1 is up, line protocol is up
Hardware is GT96K Serial
Internet address is 10.10.0.1/30
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
459 packets input, 44002 bytes, 0 no buffer
Received 57 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
393 packets output, 37028 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
11 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

O*E2 0.0.0.0/0 [110/1] via 10.10.0.2, 00:07:36, Serial0/1/1
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
C 10.10.0.0/30 is directly connected, Serial0/1/1
L 10.10.0.1/32 is directly connected, Serial0/1/1
C 10.10.0.4/30 is directly connected, Serial0/1/0
L 10.10.0.5/32 is directly connected, Serial0/1/0
O 10.10.0.8/30 [110/128] via 10.10.0.6, 00:05:53, Serial0/1/0
[110/128] via 10.10.0.2, 00:06:13, Serial0/1/1
172.16.0.0/16 is variably subnetted, 12 subnets, 2 masks
O 172.16.0.1/32 [110/65] via 10.10.0.6, 00:06:43, Serial0/1/0
O 172.16.1.1/32 [110/65] via 10.10.0.6, 00:06:43, Serial0/1/0
O 172.16.2.1/32 [110/65] via 10.10.0.6, 00:06:43, Serial0/1/0
C 172.16.3.0/24 is directly connected, Loopback0
L 172.16.3.1/32 is directly connected, Loopback0
C 172.16.4.0/24 is directly connected, Loopback1
L 172.16.4.1/32 is directly connected, Loopback1
C 172.16.5.0/24 is directly connected, Loopback2
L 172.16.5.1/32 is directly connected, Loopback2
O 172.16.6.1/32 [110/65] via 10.10.0.2, 00:07:41, Serial0/1/1
O 172.16.7.1/32 [110/65] via 10.10.0.2, 00:07:41, Serial0/1/1
O 172.16.8.1/32 [110/65] via 10.10.0.2, 00:07:41, Serial0/1/1
O 192.168.0.0/24 [110/65] via 10.10.0.2, 00:07:36, Serial0/1/1
O 192.168.3.0/24 [110/65] via 10.10.0.2, 00:06:03, Serial0/1/1
O 192.168.4.0/24 [110/66] via 10.10.0.2, 00:06:03, Serial0/1/1
O 192.168.20.0/24 [110/66] via 10.10.0.2, 00:06:03, Serial0/1/1
O 192.168.30.0/24 [110/66] via 10.10.0.2, 00:06:03, Serial0/1/1

R3

R3#show int s0/3/0
Serial0/3/0 is up, line protocol is up
Hardware is GT96K Serial
Internet address is 10.10.0.9/30
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
110 packets input, 9081 bytes, 0 no buffer
Received 54 broadcasts (0 IP multicasts)
22 runts, 17 giants, 0 throttles
9939 input errors, 1878 CRC, 1522 frame, 2828 overrun, 0 ignored, 3694 abort
268 packets output, 23354 bytes, 0 underruns
0 output errors, 0 collisions, 11 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

R3#show int s0/3/1
Serial0/3/1 is up, line protocol is up
Hardware is GT96K Serial
Internet address is 10.10.0.6/30
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:00, output 00:00:05, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 1000 bits/sec, 3 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
343 packets input, 32465 bytes, 0 no buffer
Received 75 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
170 packets output, 13641 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
4 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
13 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

O*E2 0.0.0.0/0 [110/1] via 10.10.0.10, 00:08:03, Serial0/3/0
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O 10.10.0.0/30 [110/128] via 10.10.0.10, 00:08:03, Serial0/3/0
[110/128] via 10.10.0.5, 00:08:51, Serial0/3/1
C 10.10.0.4/30 is directly connected, Serial0/3/1
L 10.10.0.6/32 is directly connected, Serial0/3/1
C 10.10.0.8/30 is directly connected, Serial0/3/0
L 10.10.0.9/32 is directly connected, Serial0/3/0
172.16.0.0/16 is variably subnetted, 12 subnets, 2 masks
C 172.16.0.0/24 is directly connected, Loopback0
L 172.16.0.1/32 is directly connected, Loopback0
C 172.16.1.0/24 is directly connected, Loopback1
L 172.16.1.1/32 is directly connected, Loopback1
C 172.16.2.0/24 is directly connected, Loopback2
L 172.16.2.1/32 is directly connected, Loopback2
O 172.16.3.1/32 [110/65] via 10.10.0.5, 00:08:51, Serial0/3/1
O 172.16.4.1/32 [110/65] via 10.10.0.5, 00:08:51, Serial0/3/1
O 172.16.5.1/32 [110/65] via 10.10.0.5, 00:08:51, Serial0/3/1
O 172.16.6.1/32 [110/65] via 10.10.0.10, 00:08:03, Serial0/3/0
O 172.16.7.1/32 [110/65] via 10.10.0.10, 00:08:03, Serial0/3/0
O 172.16.8.1/32 [110/65] via 10.10.0.10, 00:08:03, Serial0/3/0
O 192.168.0.0/24 [110/65] via 10.10.0.10, 00:08:03, Serial0/3/0
O 192.168.3.0/24 [110/65] via 10.10.0.10, 00:08:03, Serial0/3/0
O 192.168.4.0/24 [110/66] via 10.10.0.10, 00:08:03, Serial0/3/0
O 192.168.20.0/24 [110/66] via 10.10.0.10, 00:08:03, Serial0/3/0
O 192.168.30.0/24 [110/66] via 10.10.0.10, 00:08:03, Serial0/3/0

Thanks for the quick answer and let me know if im missing anything or its unclear.

[Chris.k88]

Adding the below as it might make it more clear for how my Gig ports are set up

R1

interface GigabitEthernet0/0
ip address dhcp
ip nat outside
ip nat enable
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address 192.168.3.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto

[Meheretab Mengistu · Accepted solution]

Thank you for the additional information! I do not see ip nat inside statement on any of the serial interfaces; I saw it only on Gig0/1. Please add the command to the serial links of the router connected to the ISP and check again.

Please do not forget to rate helpful replies and to accept correct answers.

HTH,

Meheretab

[Chris.k88]

You're amazing, added the command and now R2 and R3 can reach the internet!

R1 the ISP router can still not ping out to the internet though.

[Meheretab Mengistu]

On R1, you need to remove ip nat enable command as it is not needed when you run ip nat inside/outside commands. Your command should be as follows:

On R1:

interface GigabitEthernet0/0
ip address dhcp
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto

If you still have a problem, please modify the access-list. Generally, it is better to add the subnets you want to NAT only.

Please do not forget to rate helpful replies and to accept correct answers.

HTH,

Meheretab

[Chris.k88]

Removed the ip nat enable, no change

adjusted my access list for the specific subnets, no change

ip access-list standard NAT
permit 192.168.0.0 0.0.255.255
permit 10.10.0.0 0.0.255.255
permit 172.16.0.0 0.0.255.255

If I do a regular ping from R1 there is no response. When i do extended ping from G0/1 interface (192.168.3.1) it goes through.

I apologise for all the bugging this problem has been getting to me for a few days, it did help resolve my NTP issue on R2 and R3 once they could reach the internet like I thought it would so VERY much appreciating the help. 

[Meheretab Mengistu]

I saw that the access-list still includes IP address of the outside interface (192.168.0.0/24). Please modify the access-list as follows:

ip access-list standard NAT

  deny 192.168.0.0 0.0.0.255   ! Please make sure this one is on the top of the list.
  permit 192.168.0.0 0.0.255.255
  permit 10.10.0.0 0.0.255.255
  permit 172.16.0.0 0.0.255.255

If you still have a problem, please share the output of show ip nat trans .

HTH,

Meheretab

[Chris.k88]

That did it!! I'm little curious though as I don't understand why it works if you wouldn't mind explaining?

[Meheretab Mengistu · Accepted solution]

With the original ACL you used (either the one with permit any or the second one including the ip nat outside interface IP address), you were performing NAT on all outgoing traffics. When you run ping from R1 (eg. ping 8.8.8.8) what you are actually running is ping 8.8.8.8 source gi0/0 (or ping 8.8.8.8 source 192.168.0.108). It was not working as it was attempting to NAT the icmp traffic which was sourced from the outgoing interface. We resolved the issue by removing the outgoing interface from being NATed (we denied 192.168.0.0/24 network from considering to be NATed).

Note that: when you run ping or traceroute on a router, the router uses the ip address of the outgoing interface as the source ip address. 

For more information on NAT order of operations, please look at the following link:- https://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/6209-5.html

HTH,

Meheretab

[Chris.k88]

I get it now.

Thank you very much for all your help.

Again much appreciated!!

[Meheretab Mengistu]

You are very welcome!