2911 not passing LAN traffic to Public interface

[mike.welker]

Perhaps it's because it's late at night, but I am becoming more frustrated with what should be a simple configuration for a proof-of-concept for internet bandwidth offloading.  This is only a temporary test, and as such, there is not much security involved. 

We have a 2911 Router running 15.0(1)M4.

• G 0/0 is our LAN interface, and it has three subinterfaces
  • G0/0.1 is our data LAN, and the gateway for our Windows machines.  This is the interface this question concerns.
  • G0/0.23 is a separate LAN for various equipment
  • G0/0.192 is another LAN for equipment
• G 0/1 is connected to the internet, and has a public address.
• S 0/0/0 is a T1 PPP, connected to our core data center
• S 0/1/0 is a backup T1 PPP, again, connected to our core data center.
• There are three static routes entered:
  • ip route 0.0.0.0 0.0.0.0 10.12.1.1 100 This is the first PPP
  • ip route 0.0.0.0 0.0.0.0 10.13.1.1 200 This is the secondary PPP
  • ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx 255 It currently has a cost of 255 while i figure this one out. xxx.xxx.xxx.xxx represents the cable company gateway, which I can ping properly.  I've also used "gigabitethernet 0/1" in place of the next hop ip with the same results.

The public interface is properly connected, and can ping it's next hop (the cable company gateway).  When I change the static route for gigabitethernet 0/1 to a cost of "0", the router can properly ping DNS names, such as google.com through the public interface. 

However, devices on the data LAN cannot reach any public addresses except for the router's public interface, let alone DNS names (I am using 8.8.8.8 as my test IP).  If I revert the cost back to 255, making the PPP the gateway of last resort, these devices can again connect. (they travel through the PPP to our Data center's internet) 

This confuses me.  If our server, on the same LAN as the router can ping the public interface (it's definitley not leaving the 2911, as latency is less than 1ms), and the router itself can ping outside addresses, what is preventing the router's public interface from passing traffic to the internet from any source other than itself?  I have attached our running config in the hopes that there is something obvious I'm missing (the public ip addresses have been changed so they are not exposed).  I simply want clients on our 10.23.0.0 LAN to get to the internet via the public interface of the local router, and still connect to corporate resources using the PPP links.

MAS_2911#sho run
Building configuration...
Current configuration : 5666 bytes
!
! Last configuration change at 01:47:50 eastern Sat Sep 24 2011 by redacted
! NVRAM config last updated at 15:15:05 eastern Wed Aug 24 2011 by redacted
!
version 15.0
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname MAS_2911
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 warnings
enable secret 5 redacted
!
no aaa new-model
!
!
!
clock timezone eastern -5
clock summer-time eastern recurring
!
no ipv6 cef
ip source-route
ip cef
!
!
!
!
ip domain name redacted
ip name-server 10.23.1.48
ip name-server 10.20.1.48
!
multilink bundle-name authenticated
!
!
!
crypto pki trustpoint TP-self-signed-redacted
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-redacted
 revocation-check none
 rsakeypair TP-self-signed-redacted
!
!
!
crypto pki trustpoint TP-self-signed-redacted
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-redacted
 revocation-check none
 rsakeypair TP-self-signed-redacted
!
!
crypto pki certificate chain TP-self-signed-redacted
 certificate self-signed 01
redacted
        quit
license udi pid CISCO2911/K9 sn FTX1513AHL6
!
!
username redacted
!
redundancy
!
!
!
!
!
!
!
!
!
interface GigabitEthernet0/0
 description redacted LAN
 no ip address
 ip flow ingress
 ip flow egress
 no ip route-cache cef
 duplex full
 speed 1000
 !
!
interface GigabitEthernet0/0.1
 description Data VLAN
 encapsulation dot1Q 1 native
 ip address 10.23.1.1 255.255.0.0
 ip flow ingress
!
interface GigabitEthernet0/0.23
 description PLC VLAN 23
 encapsulation dot1Q 23
 ip address 172.23.1.1 255.255.240.0
 ip access-group 10 out
 ip flow ingress
!
interface GigabitEthernet0/0.192
 description B VLAN
 encapsulation dot1Q 192
 ip address 192.168.0.254 255.255.255.0
 ip access-group 10 out
 ip flow ingress
!
interface GigabitEthernet0/1
 description Connection to Internet
 ip address xxx.xxx.xxx.xxx 255.255.254.0
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
 !
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
 !
!
interface Serial0/0/0
 description Link to Redacted
 ip address 10.13.1.2 255.255.0.0
 ip flow ingress
 ip flow egress
 no clock rate 2000000
 !
!
interface Serial0/1/0
 description backup to Redacted
 ip address 10.12.1.2 255.255.0.0
 ip flow ingress
 ip flow egress
 no clock rate 2000000
 !
!
interface Cellular0/2/0
 no ip address
 encapsulation ppp
 !
!
!
router eigrp 100
 maximum-paths 2
 network 10.0.0.0
 network 172.23.0.0 0.0.15.255
 network 192.168.0.0
 no eigrp log-neighbor-changes
!
ip default-gateway 10.23.1.1
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip flow-cache timeout active 1
ip flow-export version 5
ip flow-export destination 10.20.1.39 2055
!
ip route 0.0.0.0 0.0.0.0 10.12.1.1 100
ip route 0.0.0.0 0.0.0.0 10.13.1.1 200
ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx 255
ip route 10.20.1.19 255.255.255.255 10.12.1.1
!
ip access-list extended all
 permit ip any any
!
!
!
!
!
!
snmp-server community redacted RO
snmp-server location redacted
snmp-server contact SANT
!
control-plane
 !
!
banner exec ^C ^C
banner login ^C^C
banner motd ^CC
        <<<<<<< ----------- redacted ------------- >>>>>>>>
<<<<<<<<<<<<<<<<<<<< ---------WARNING---------- >>>>>>>>>>>>>>>>>>>>>>
redacted
<<<<<<<<<<<<<<<<<<<< ---------WARNING---------- >>>>>>>>>>>>>>>>>>>>>>
^C
!
line con 0
 exec-timeout 30 0
 password 7 redacted
 logging synchronous
 login
line aux 0
line 0/2/0
 no exec
 rxspeed 3100000
 txspeed 1800000
line vty 0 4
 access-class 23 in
 exec-timeout 0 0
 password 7 redacted
 logging synchronous
 login local
 transport input telnet ssh
line vty 5 15
 access-class 23 in
 exec-timeout 0 0
 password 7 redacted
 logging synchronous
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
ntp master
ntp server 10.20.1.251 prefer
ntp server 10.20.1.252
end
MAS_2911#

Does anyone see what I'm missing? I've also manually added the public network to eigrp to no avail.  I've since removed it because it shouldn't be something this frustrating, and I don't want to make numerous changes without validation.

[cadet alain]

Hi,

an AD of 255 means it is not a believable route so it won't get inserted into the routing table.

All static routes have an AD of 1 and if you've got a superior AD then you're doing a floating static route that will be inserted in the routing table only if the inferior AD route fails but here to get inserted you would have your 2 PPP links to be down.

If you let it with the default AD of 1 then all routes will be installed and the switching process which is CEF by default will take care of the load balancing which is per src-dst IP pair.

If you want some traffic to take a different path than what the routing table says then you'll have to do Policy-based routing(PBR).

Here is an example:

access-list 101 permit tcp 10.23.0.0 0.0.0.255 any eq http

route-map NAT

match ip address 101

match interface G0/1

route-map WEB

match ip address 101

set ip next-hop xxxx   where xxxx is internet default gateway

int g0/1

ip nat outside

int g0/0.1

ip nat inside

ip policy route-map WEB

ip nat inside route-map NAT int g0/1 overload

You can get rid of the EIGRP if you haven't got any other router inside and the ip default-gateway can be deleted because it won't be used if you're routing only if your router has ip routing disabled which is not your case.

Regards.

Alain.

[mike.welker]

Thanks Cadet.  I should have been a little more clear.  The current config of the static route only has a cost of 255 so all traffic will still travel through the PPP while I troubleshoot.  My initial configuration had the following route statements:

ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx - where xxx.xxx.xxx.xxx is the internet default gateway

ip route 0.0.0.0 0.0.0.0 10.12.1.1 100

ip route 0.0.0.0 0.0.0.0 10.13.1.1 200

What I've found with this configuration is that when G0/1 is un-shut, the first route above is indeed inserted into the routing table as the gateway of last resort.  The router can ping the internet default gateway, as well as any other internet destination.  However, internal clients can ping no further than G0/1's IP address.  If I shut down G0/1, the second route, using the PPP properly inserts, and clients regain internet access through our core data center. 

What is confusing to me is that I can ping the public interface of the router from a client, and the router can ping public addresses.  What is preventing a client from pinging past G0/1?  Perhaps a NAT issue?

[cadet alain]

Hi,

Yes on the internet the private addresses are not routeable so you need NAT.

Regards.

Alain.

[mike.welker]

I'm sorry Cadet, you've been helpful, but I am still lost.  I've updated my configuration to the following (truncated so it's easier reading):

!
no ipv6 cef
ip source-route
ip cef
!
!
redundancy
!
!
interface GigabitEthernet0/0
 description LAN
 no ip address
 ip flow ingress
 ip flow egress
 no ip route-cache cef
 duplex full
 speed 1000
 !
!
interface GigabitEthernet0/0.1
 description Data VLAN
 encapsulation dot1Q 1 native
 ip address 10.23.1.1 255.255.0.0
 ip flow ingress
 ip nat inside
 ip virtual-reassembly
!
!
interface GigabitEthernet0/1
 description Connection to Internet
 ip address xxx.xxx.xxx.xxx 255.255.254.0
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
 !
!
!
!
!
router eigrp 100
 maximum-paths 2
 network 10.0.0.0
 network 172.23.0.0 0.0.15.255
 network 192.168.0.0
 no eigrp log-neighbor-changes
!
ip default-gateway 10.23.1.1
ip forward-protocol nd
!
!
ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx
ip route 0.0.0.0 0.0.0.0 10.12.1.1 100
ip route 0.0.0.0 0.0.0.0 10.13.1.1 200
ip route 10.20.1.19 255.255.255.255 10.12.1.1
!
ip access-list extended all
 permit ip any any
!
access-list 101 remark Used for NAT of internal IPs
access-list 101 permit tcp 10.23.0.0 0.0.255.255 any
!
!
!
!
route-map WEB permit 10
 match ip address 101
 set ip next-hop xxx.xxx.xxx.xxx
!
route-map NAT permit 10
 match ip address 101
 match interface GigabitEthernet0/1
MAS_2911#

I still have the same issue with the above.  Here is a sample of my routing table with the above configuration.  I've bolded the public routes:

MAS_2911#sho ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is xxx.xxx.xxx.xxx to network 0.0.0.0
S*    0.0.0.0/0 [1/0] via xxx.xxx.xxx.xxx
      10.0.0.0/8 is variably subnetted, 16 subnets, 2 masks
D        10.1.0.0/16 [90/2684416] via 10.13.1.1, 11:41:30, Serial0/0/0
                     [90/2684416] via 10.12.1.1, 11:41:30, Serial0/1/0
D        10.10.0.0/16 [90/2681856] via 10.13.1.1, 11:41:30, Serial0/0/0
D        10.11.0.0/16 [90/2681856] via 10.12.1.1, 11:41:30, Serial0/1/0
C        10.12.0.0/16 is directly connected, Serial0/1/0
L        10.12.1.2/32 is directly connected, Serial0/1/0
C        10.13.0.0/16 is directly connected, Serial0/0/0
L        10.13.1.2/32 is directly connected, Serial0/0/0
D        10.14.0.0/16 [90/2681856] via 10.13.1.1, 11:41:30, Serial0/0/0
D        10.18.0.0/16 [90/2681856] via 10.13.1.1, 11:41:30, Serial0/0/0
D        10.19.0.0/16 [90/2681856] via 10.12.1.1, 11:41:30, Serial0/1/0
D        10.20.0.0/16 [90/2170112] via 10.13.1.1, 11:41:31, Serial0/0/0
S        10.20.1.19/32 [1/0] via 10.12.1.1
D        10.22.0.0/16 [90/2684416] via 10.13.1.1, 11:41:31, Serial0/0/0
                      [90/2684416] via 10.12.1.1, 11:41:31, Serial0/1/0
C        10.23.0.0/16 is directly connected, GigabitEthernet0/0.1
L        10.23.1.1/32 is directly connected, GigabitEthernet0/0.1
D        10.24.0.0/16 [90/2684416] via 10.13.1.1, 11:41:31, Serial0/0/0
      xxx.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        xxx.xxx.xxx.0/23 is directly connected, GigabitEthernet0/1
L        xxx.xxx.xxx.xxx/32 is directly connected, GigabitEthernet0/1
D     172.21.0.0/16 [90/2682112] via 10.13.1.1, 11:41:31, Serial0/0/0
                    [90/2682112] via 10.12.1.1, 11:41:31, Serial0/1/0
      172.23.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.23.0.0/20 is directly connected, GigabitEthernet0/0.23
L        172.23.1.1/32 is directly connected, GigabitEthernet0/0.23
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, GigabitEthernet0/0.192
L        192.168.0.254/32 is directly connected, GigabitEthernet0/0.192
MAS_2911#

So, given the routing table, and using IP only at this point, here's what is happening:

• From a server on the local LAN
  • I can ping corporate resources (anything on the 10.xxx.xxx.xxx network)
  • I can ping the public address of the router on G0/1
  • I cannot ping the public gateway of the router
• From the router
  • I can ping corporate resources
  • I can ping the public gateway for the internet
  • I can ping public IP addresses, such as 8.8.8.8

So, with the above, I'm pretty positive it's a NAT issue.  However, because the routing table gives me full access to corporate resources via EIGRP, and the gateway of last reost is pointed to the public side, i should not have to do policy based routing, correct?  I should only need to tell the interfaces that they are either the inside NAT or the outside NAT.

I think I'm having a brain fart with this one.  Thank you so much for all of your help.  I may go as far as opening a TAC case and having a fresh set of eyes look at it.

[cadet alain · Accepted solution]

Hi,

Forget about PBR for now. Yes it is a nat problem because you must tell which interface is inside and which is outside but if you don't tell the router which subnet should be natted and what must be the natted address it won't work.

So here if you want nat to work:

1) make an ACL permitting the subnet to be natted

    access-list  1 permit 10.23.0.0 0.0.255.255

2) tell the router how this subnet should be natted

   ip nat inside source list 1 interface g0/1 overload

Once this is done it should be ok.

Then we will discuss about choosing which path some traffic should use and failover scenario.

Regards.

Alain

[mike.welker]

That did the trick, Alain.  Thanks a million!  I knew it was something simple, and it was the source list that was missing.

All clients on the data network can reach both corporate resources, and the internet using the correct internet connection on site.  If I am correct, i already have failover built in, because if the internet connection on site fails, the secondary static route of 0.0.0.0 0.0.0.0 10.12.1.1 100 will take over, and clients can access the internet using that method.