Routing help needed ASAP PLEASE!!!

Michael Durham · ‎09-30-2015

I have been fighting this for a week plus now and I know I should know this but I just cannot seem to get it all working. Here is my layout.

Let's tart with what is working correctly, from Building 3 their Internet normally goes out fa0/1 to a CradelPoint router (cannot change this) and it works fine. But if the CradelPoint goes down the SLA on the switch routes them out fa0/48 to the backup Internet connection and this too works. No problems here!

Building 1 and two normally use the 2851 router's Internet connection connected to gi0/0 to a TP-Link router (cannot be changed) and it too works. Except to traceroute command does not work. I can ping out the Internet just fine but traceroute returns all *'s; however, a week ago it did work. Everyone in all 3 buildings can use this Internet and tracert works from the PC's just no traceroute from any switch or router.

I need for buildings 1 and 2 to be able to use the backup Internet source in building 3 should the Internet connection to the 2851 go down.This is where the problem comes in.

When the Internet connection on the 2851 goes down the SLA changes the route to go out building 3's Internet and if you issue a ping 4.2.2.2 source 192.168.0.253 it works!!! Even traceroute 4.2.2.2 source 192.168.0.253 works on this router. BUT if you issue a ping 4.2.2.2 source 192.168.69.1, no connection. Therefore no one in buildings 1 and 2 have backup Internet. I just cannot figure out why VLAN2 is not talking to VLAN's 50,69,110,125, and 200. Any suggestions?

ROUTER 2851 CONFIG:

Current configuration : 18615 bytes
!
! Last configuration change at 17:08:23 DST Wed Sep 30 2015 by mdurham
! NVRAM config last updated at 17:08:36 DST Wed Sep 30 2015 by mdurham
! NVRAM config last updated at 17:08:36 DST Wed Sep 30 2015 by mdurham
version 15.1
service timestamps debug uptime
service timestamps log datetime msec localtime year
service password-encryption
service sequence-numbers
!
hostname CME_Router
!
boot-start-marker
boot-end-marker
!
!
logging buffered 10000000
enable secret 5 Cisco
!
no aaa new-model
!
clock timezone Eastern -5 0
clock summer-time DST recurring
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-2516279958
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-2516279958
revocation-check none
rsakeypair TP-self-signed-2516279958
!
!
crypto pki certificate chain TP-self-signed-2516279958
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 32353136 32373939 3538301E 170D3133 30373233 30373030
31355A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D32 35313632
37393935 3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100A322 EE57B83E 4C7B7947 8EE8EF6D 931234DB 41A4D133 19D8D297 32FF0E5F
10B94981 E7A1EA06 76CD851B FA24BDA8 561F4CC6 22C3B73B B0742465 571E07C9
04EE1661 7FC5B487 B6686CBB 22B49313 238DF90C 3795649E FA8CAD50 C0CE6A9D
E11FE11E 66870AC2 1492AE74 BBA1E419 1A7FE7F0 C88A8543 5CC6316C 2600101B
A32B0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
551D2304 18301680 1402DCB2 F8C6A62B 687347AB DEA0875E ECB7063E 8C301D06
03551D0E 04160414 02DCB2F8 C6A62B68 7347ABDE A0875EEC B7063E8C 300D0609
2A864886 F70D0101 05050003 8181002E 9E5AC99B CCCF28EB 9D517ECD EA130719
EB97B0CA 09CA4AA9 5D88C9C0 FFAE75FE 09B78CD5 3023BC39 B98E4B1F D3B72C82
FDCB8BA1 620A9433 3ED237D6 CFCB8D1E 67BD4232 FECF61EB 37499502 BB3F2899
E77260E9 985F9938 C47E7A09 37A07302 3C851D7A DAEF35A5 495E0220 C3F17AFB
B68AE9D3 F0CFE894 7E8DBC39 ED077B
quit
dot11 syslog
ip source-route
!
!
ip cef
!
ip dhcp excluded-address 192.168.69.1 192.168.69.240
ip dhcp excluded-address 192.168.200.1 192.168.200.240
ip dhcp excluded-address 10.110.0.1 10.110.0.20
ip dhcp excluded-address 192.168.125.1 192.168.125.9
!
ip dhcp pool AirPark-WiFi
network 192.168.200.0 255.255.255.0
default-router 192.168.200.1
domain-name mtd.home
dns-server 4.2.2.2
lease 0 12
!
ip dhcp pool Users
network 192.168.69.0 255.255.255.0
default-router 192.168.69.1
domain-name mtd.home
dns-server 4.2.2.2
option 150 ip 10.110.0.1
lease 0 12
!
ip dhcp pool Voice
network 10.110.0.0 255.255.255.0
default-router 10.110.0.1
option 150 ip 10.110.0.1
dns-server 4.2.2.2
domain-name mtd.home
lease 0 12
!
ip dhcp pool TV
network 192.168.125.0 255.255.255.0
default-router 192.168.125.1
domain-name mtd.home
dns-server 4.2.2.2
lease 0 12
!
ip dhcp pool Reserved69
host 192.168.69.101 255.255.255.0
hardware-address 0800.091a.8987
default-router 192.168.69.1
dns-server 4.2.2.2
domain-name mtd.home
!
ip dhcp pool WirelessRES
host 192.168.200.252 255.255.255.0
hardware-address 001d.e04d.c323
default-router 192.168.200.1
dns-server 4.2.2.2
domain-name mtd.home
!
!
ip domain lookup source-interface GigabitEthernet0/0
ip domain name mtd.home
ip name-server 4.2.2.2
ip name-server 8.8.8.8
no ipv6 cef
!
multilink bundle-name authenticated
!
!
password encryption aes
!
voice-card 0
!
!
license udi pid CISCO2851 sn FTX1331AJZF
dial-control-mib retain-timer 10080
dial-control-mib max-size 500
username mdurham privilege 15 password 7 Cisco
!
redundancy
!
!
track 10 ip sla 1 reachability
delay down 1 up 1
!
track 20 ip sla 2 reachability
delay down 1 up 1
gw-accounting syslog
!
!
!
interface GigabitEthernet0/0
ip address 192.168.254.1 255.255.255.0
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface Service-Engine0/1
description "CUE-AIM Voicemail Module"
ip unnumbered GigabitEthernet0/1.110
ip nat inside
ip virtual-reassembly in
service-module ip address 10.110.0.2 255.255.255.0
service-module ip default-gateway 10.110.0.1
!
interface GigabitEthernet0/1
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0/1.2
description Alt Internet Conntection via Sommer's
encapsulation dot1Q 2
ip address 192.168.0.253 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.50
description "VMWare Server"
encapsulation dot1Q 50
ip address 192.168.50.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.69
description "Data Network"
encapsulation dot1Q 69 native
ip address 192.168.69.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.110
description "Voice Network"
encapsulation dot1Q 110
ip address 10.110.0.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.125
description "TV & Media Network"
encapsulation dot1Q 125
ip address 192.168.125.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.200
description "Guest User Network"
encapsulation dot1Q 200
ip address 192.168.200.1 255.255.255.0
ip access-group 100 in
ip nat inside
ip virtual-reassembly in
!
!
router eigrp 1577
network 10.110.0.0 0.0.0.255
network 192.168.0.0
network 192.168.50.0
network 192.168.69.0
network 192.168.125.0
network 192.168.200.0
network 192.168.254.0
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
!
ip route 0.0.0.0 0.0.0.0 192.168.254.254 track 10
ip route 0.0.0.0 0.0.0.0 192.168.0.254 10 track 20
ip route 10.110.0.2 255.255.255.255 Service-Engine0/1
!
ip sla 1
icmp-echo 192.168.254.254 source-ip 192.168.254.1
threshold 750
timeout 900
frequency 1
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 192.168.0.254 source-ip 192.168.0.253
threshold 750
timeout 900
frequency 1
ip sla schedule 2 life forever start-time now
logging esm config

From the 2851 router's cli:

Gateway of last resort is 192.168.0.254 to network 0.0.0.0

S*    0.0.0.0/0 [10/0] via 192.168.0.254
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.110.0.0/24 is directly connected, GigabitEthernet0/1.110
L        10.110.0.1/32 is directly connected, GigabitEthernet0/1.110
S        10.110.0.2/32 is directly connected, Service-Engine0/1
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, GigabitEthernet0/1.2
L        192.168.0.253/32 is directly connected, GigabitEthernet0/1.2
      192.168.50.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.50.0/24 is directly connected, GigabitEthernet0/1.50
L        192.168.50.1/32 is directly connected, GigabitEthernet0/1.50
      192.168.69.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.69.0/24 is directly connected, GigabitEthernet0/1.69
L        192.168.69.1/32 is directly connected, GigabitEthernet0/1.69
      192.168.125.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.125.0/24 is directly connected, GigabitEthernet0/1.125
L        192.168.125.1/32 is directly connected, GigabitEthernet0/1.125
      192.168.200.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.200.0/24 is directly connected, GigabitEthernet0/1.200
L        192.168.200.1/32 is directly connected, GigabitEthernet0/1.200

traceroute 4.2.2.2 source 192.168.0.253
Type escape sequence to abort.
Tracing the route to b.resolvers.Level3.net (4.2.2.2)
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.0.254 4 msec 0 msec 0 msec
2 192.168.0.1 0 msec 0 msec 0 msec
3 66.174.39.85 44 msec 40 msec 40 msec
4 69.83.39.164 36 msec 36 msec 44 msec
5 69.83.39.115 40 msec 44 msec 40 msec
6 69.83.39.194 28 msec 44 msec 36 msec
7 69.83.33.176 40 msec 40 msec 40 msec

traceroute 4.2.2.2 source 192.168.69.1
Type escape sequence to abort.
Tracing the route to b.resolvers.Level3.net (4.2.2.2)
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.0.254 4 msec 0 msec 4 msec
2 192.168.0.1 0 msec 0 msec 0 msec
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *

I need to get this solved this week,

Thank You in advance!

Michael

Jon Marshall · ‎09-30-2015

Michael

Ignore that, your traceroute shows it does have routes back.

Have you setup NAT on the Cradlepoint for the other subnets ?

Jon

Michael Durham · ‎09-30-2015

Yes, forgot to add these pics:

SWITCH 3 info:

sh ip route

Gateway of last resort is 192.168.0.1 to network 0.0.0.0

D    192.168.125.0/24 [90/28416] via 192.168.0.253, 20:57:25, Vlan2
D    192.168.200.0/24 [90/28416] via 192.168.0.253, 20:57:25, Vlan2
     10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
D       10.110.0.0/24 [90/28416] via 192.168.0.253, 20:57:25, Vlan2
D       10.110.0.2/32 [90/28416] via 192.168.0.253, 20:57:25, Vlan2
C    192.168.0.0/24 is directly connected, Vlan2
D    192.168.254.0/24 [90/28416] via 192.168.0.253, 00:39:21, Vlan2
D    192.168.50.0/24 [90/28416] via 192.168.0.253, 20:57:25, Vlan2
D    192.168.69.0/24 [90/28416] via 192.168.0.253, 20:23:00, Vlan2
S*   0.0.0.0/0 [1/0] via 192.168.0.1

!
aaa session-id common
clock timezone Eastern -5
clock summer-time est recurring
!
track 10 rtr 1 reachability
delay down 1 up 1
!
track 20 rtr 2 reachability
delay down 1 up 1
ip subnet-zero
ip routing
no ip domain-lookup
ip dhcp excluded-address 192.168.0.1 192.168.0.150
ip dhcp excluded-address 192.168.0.200 192.168.0.254
!
ip dhcp pool Sommer
   network 192.168.0.0 255.255.255.0
   default-router 192.168.0.254
   dns-server 4.2.2.2
   lease 0 12
!
ip sla responder
ip sla responder udp-echo ipaddress 172.16.31.1 port 5000
ip sla responder udp-echo ipaddress 192.168.0.254 port 5000
ip sla 1
icmp-echo 192.168.0.1 source-ip 192.168.0.254
timeout 500
frequency 1
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 192.168.0.253 source-ip 192.168.0.254
timeout 500
frequency 1
ip sla schedule 2 life forever start-time now
vtp mode transparent
!
password encryption aes
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 2
name Sommer's
!
!
interface FastEthernet0/1
description Sommer's connection to Verizon 4G via CradlePoint Router
switchport access vlan 2
switchport mode access
switchport nonegotiate
spanning-tree portfast
!
interface FastEthernet0/2
switchport access vlan 2
switchport mode access
switchport nonegotiate
spanning-tree portfast
!

interface FastEthernet0/48
description WHITE - Connection to Durham's Garage port fa0/23 (172.16.31.3)
switchport access vlan 2
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 1,2,69,1002-1005
switchport mode trunk
switchport nonegotiate
spanning-tree portfast
!
interface GigabitEthernet0/1
switchport mode dynamic desirable
!
interface GigabitEthernet0/2
switchport mode dynamic desirable
!
interface Vlan1
no ip address
!
interface Vlan2
ip address 192.168.0.254 255.255.255.0
!
!
router eigrp 1577
network 172.16.31.0 0.0.0.7
network 192.168.0.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.0.1 track 10
ip route 0.0.0.0 0.0.0.0 192.168.0.253 10 track 20
ip http server
!
!
!
control-plane
!

Pings and traceroutes go out fa0/1 normally and out fa0/48 when fa0/1 is down.

Jon Marshall · ‎09-30-2015

Michael

I actually modified my original response so perhaps you didn't see it.

I can see it's not the routing but have you added NAT rules for those subnets ?

Jon

Michael Durham · ‎09-30-2015

I have not done that before. One note, a few years back I had this working using a wireless bridge between the two buildings and we did not even have to add my networks in building 3's router. But I do not remember if it was using the CradelPoint then.

There is a ip nat inside configured on the gi0/1.x sub interfaces and ip nat outside on the gi0/0 interface.

Just so you know, on buildings 1 & 2 the Internet is provided by a Verizon 4G mifi to the TP-Link bridge/router. In building 3 it too is Verizon 4G to the CradelPoint via an attached CradlePoint 4G modem

Michael Durham · ‎09-30-2015

If we could get DSL or cable here WE WOULD. This is our ONLY option because both Verizon plans have unlimited data and Sat is too slow!

Jon Marshall · ‎09-30-2015

I don't think it is a routing issue because you can traceroute to the Cradlepoint.

It is definitely looking like NAT.

If the above is from the Cradlepoint you can see it is doing NAT for the 192.168.0.0/24 subnet which is why that works.

But it doesn't mention 192.168.69.x so those IPs are not being translated which means they won't be routed on the internet hence you don't get a response.

What we could have tried is to NAT the 192.168.69.x IPs to the 192.168.0.253 IP on your 2851 router but that would mean adding "ip nat outside" to the gi0/1.2 subinterface but unfortunately you already have "ip nat inside" on that for your other internet connection and you need this otherwise that wouldn't work.

I know that some routers support NAT without having to add "ip nat inside/outside" to interfaces but I have never used it so can't say whether it would be a solution or not.

However the first thing to do is see if you can add NAT rules to your Cradlepoint router.

Jon

Michael Durham · ‎09-30-2015

The CP cannot do NAT rules. For some unknown reason even though I have added the routes to it, it does not seem to be using them.

What if I put a real router between the CP and the 3550? One side could be the WAN and the other the LAN. Would that give us more options? I have a few spare 2811's laying around

Jon Marshall · ‎09-30-2015

It is using the routes because your traceroute shows it is.

The issue is 192.168.69.x is a private IP range ie. not routable on the internet which means the Cradlepoint forwards the traffic to it's next hop but then the next hop cannot route the traffic back.

Yes, if you have a spare router that would work ie. you could NAT all traffic through it to a 192.168.0.x IP which the Cradlepoint is doing NAT for.

So you have a choice -

1) use the 192.168.0.x IP subnet between the router and the Cradlepoint but this would mean readdressing any clients using the 192.168.0.x subnet because they need a new IP subnet

or

2) keep the 192.168.0.x IP subnet for clients and use a new IP subnet between the router and the Cradlepoint.

I suspect whatever IP subnet is configured on the inside interface of the Cradlepoint, it will do NAT for so it is up to you which one you want to do.

Then you simply NAT all traffic through the router to the IP address of the interface connecting to the Cradlepoint.

You will also need to add routes to the new router.

Does this make sense ?

Jon

Michael Durham · ‎09-30-2015

I assume that you mean put NAT inside on the inside net and NAT outside on the outside net? Anything else?

Jon Marshall · ‎09-30-2015

Yes, you have to make sure that all IPs are translated to an IP that is from the same IP subnet as the inside interface of the Cradlepoint.

The other thing will be routing.

I think the easiest solution is to keep the 192.168.0.x subnet for clients and move the 192.168.0.1 IP address to the inside interface of your new router.

That way your routing on the 2851 doesn't need to change.

You would need a default route on the new router pointing to the inside interface IP of the Cradlepoint and then you also need to add routes to the new router for all the other subnets on the 2851 eg.

"ip route 192.168.69.0 255.255.255.0 192.168.0.253"

etc. for each subnet.

Or run a dynamic routing protocol between the routers.

Then you use a new IP subnet between the outside interface of your new router and the Cradlepoint device.

Like I said before I am assuming the Cradlepoint will automatically NAT the IP subnet configured on it's inside interface and the solution above relies on this.

This would be a lot less complicated than the other option I outlined.

Edit - don't forget to update the IP SLA configuration on your switch ie. you need to ping the new IP on the Cradlepoint.

Jon

Michael Durham · ‎10-01-2015

Problem solved. Before we took any additional steps as your suggested, we did a firmware update on the CradlePoint and the solved the problem. Thank you so much for your time!

Jon Marshall · ‎10-01-2015

No problem. glad you got it working.

Jon

Jon Marshall · ‎09-30-2015

edited

Jon Marshall · ‎09-30-2015

Michael

Just as a follow up to this the Cradlepoint output suggests it is also doing NAT for the 192.168.10.0/24 subnet.

Is this subnet in use ?

If not it may be that we could create a subinterface on the 2851 for that IP subnet and then NAT the traffic to a 192.168.10.x IP.

I would check if you can add NAT rules first but if not that might be a possibility although I can't say for sure as I have never used that device before so I may be misreading the output.

Jon