ISR 4321 Routing Issues

[mwgallotte]

Environment: Broadband Ethernet hand-off over 1Gbps copper link into Cisco ISR 4321 where only NATing and external routing will occur. Router connects directly to Core switch Cisco 4507R-E for distribution to closet switches. DHCP and Vlan Interfaces reside on the Core switch. Routing protocol is OSPF.

Situation: Replacing old Cisco 2600 router with new ISR 4321. Migrated bare bones old config to new device without issues. Everything looks OK.

Problem: Cannot successfully ping the ISP gateway from any device other than the new ISR 4321. All other devices on network including the core switch can only ping to our local public IP address, which is on the same subnet as the ISP's gateway. The router can ping out to the ISPs gateway and to 8.8.8.8 without any problem. The new router can also ping the core switch and the vlan interfaces. I can place the old router back in with the same configuration and everything works fine. WHAT AM I DOING WRONG??? PLEASE HELP!!! The configs are below minus all of the fluff.

ISP Gateway: 67.20.6.65 255.255.255.248

Router Configuration Cisco ISR-4321 Below:

Router#show run
Building configuration...

Current configuration : 1325 bytes
!
! Last configuration change at 19:15:06 UTC Fri Jul 8 2016
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
no aaa new-model
!
subscriber templating
multilink bundle-name authenticated
!
license udi pid ISR4321/K9 sn FDO20040Y6Z
!
spanning-tree extend system-id
!
!
redundancy
mode none
!
!
vlan internal allocation policy ascending
!
interface GigabitEthernet0/0/0
description LINK TO CISCO 4507R-E Gi1/48
ip address 172.16.1.1 255.255.255.0
ip nat inside
negotiation auto
!
interface GigabitEthernet0/0/1
description LINK TO ISP
ip address 67.20.6.66 255.255.255.248
ip nat outside
negotiation auto
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
!
interface Vlan1
no ip address
shutdown
!
router ospf 1
network 67.20.6.66 0.0.0.0 area 0
network 172.16.1.1 0.0.0.0 area 0
!
ip nat inside source list 1 interface GigabitEthernet0/0/1 overload
ip forward-protocol nd
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet0
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/1
!
access-list 1 permit 172.16.0.0 0.0.255.255
!
control-plane
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
login
!
end

Core Switch Cisco 4507R-E Configuration Below

81MDG-PN-4B144-CORE#show run
Building configuration...

Current configuration : 9426 bytes
!
! Last configuration change at 13:21:43 UTC Thu Jul 7 2016
! NVRAM config last updated at 07:15:43 UTC Fri Jul 8 2016
!
version 15.1
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service compress-config
!
hostname 81MDG-PN-4B144-CORE
!
boot-start-marker
boot-end-marker
!
!
no logging console
enable password 7 004E47530F035A0B1B270D
!
no aaa new-model
!
ip vrf mgmtVrf
!
ip dhcp excluded-address 172.16.40.1 172.16.40.10
ip dhcp excluded-address 172.16.50.1 172.16.50.100
ip dhcp excluded-address 172.16.70.1 172.16.70.50
!
ip dhcp pool Vlan10-SGSI
network 172.16.40.0 255.255.255.0
dns-server 69.89.215.5
domain-name 81mdgpn
default-router 172.16.40.1
lease 2
!
ip dhcp pool Vlan40-MDGUsers-Verizon
network 172.16.70.0 255.255.254.0
dns-server 69.89.215.5
domain-name 81mdgpn
default-router 172.16.70.1
lease 2
!
ip dhcp pool Vlan20-Genetics
network 172.16.50.0 255.255.254.0
dns-server 69.89.215.5
domain-name 81mdgpn
default-router 172.16.50.1
lease 2
!
power redundancy-mode redundant
!
spanning-tree mode pvst
spanning-tree extend system-id
!
redundancy
mode sso
!
vlan internal allocation policy ascending
!
interface FastEthernet1
ip vrf forwarding mgmtVrf
no ip address
speed auto
duplex auto
!
interface GigabitEthernet1/48
description LINK TO ROUTER GI0/0/0
no switchport
ip address 172.16.1.2 255.255.255.0
!
!
interface Vlan1
description Management
ip address 172.16.30.30 255.255.254.0
!
interface Vlan10
description SGSI Users
ip address 172.16.40.1 255.255.255.0
ip helper-address 172.16.40.1
!
interface Vlan20
description Genetics
ip address 172.16.50.1 255.255.254.0
ip helper-address 172.16.50.1
!
interface Vlan30
description Public WiFi
ip address 172.16.60.1 255.255.252.0
ip helper-address 172.16.60.1
!
interface Vlan40
description MDG Users & Verizon
ip address 172.16.70.1 255.255.254.0
ip helper-address 172.16.70.1
!
router ospf 1
network 172.16.0.0 0.0.255.255 area 0
!
no ip http server
no ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 172.16.1.1
!
!

[chrihussey]

Your default route to the ISP could be the issue. The destination should not be the ISP LAN interface but an actual IP address:

ip route 0.0.0.0 0.0.0.0 67.20.6.65

By specifying the LAN interface, a lot of things can go wrong. The router has to make constant ARP requests for all traffic flows going to the Internet and this may be getting in the way of having a proper route table and subsequently allowing the NAT operation from occurring.

[mwgallotte]

Chrihussey,

I've tried that routing entry with the same result.  Any other ideas?

[chrihussey]

I'd still change it back to only using the route with the IP address. The interface isn't a good idea.

Other than that nothing jumps out at me at this time, debugging the nat process while pinging from the 4507 from the may provide some answers. 

[mwgallotte]

Will do.  However, NAT isn't the issue.  The issue is I can't get any traffic through the router to the ISP.  From the router I can ping the ISP, from the core switch behind the router I cannot. 

[chrihussey · Accepted solution]

You can ping the ISP from the router because you are pinging them with an IP (67.20.6.66) on their network.

If NAT was working, then the traffic going through the router would be hitting the ISP with the same IP, which does not appear to be the case. If NAT isn't working (can't rule out a bug), then you may be hitting the ISP with the private IP space and would be black holed.

Aside from debugging the NAT, clear the counters on the 4321 interfaces, set the load-interval to 30, send a large extended ping to the Internet from the 4507 and look at the counters on the 4321 interfaces to determine if your seeing traffic in and out on both interfaces.

If you can get a sniffer trace of the ISP interface that may also help in providing some answers.