Email notifications stopped working after adding 2nd ISP -asymmetrical?

[Darren Spezio]

e-mail notifications and scan to e-mail from the copiers stopped working after adding second ISP using NATing, both use aspmx.l.google.com, regular e-mail communcation between users work fine. I'm guessing the routing is asymmetrical , what can I do to make it symmetrical ?

ISP was added for additional bandwidth. As soon as I remove the 2nd IP route, the email notifications and scan to email start working again.

Cisco 2900

interface GigabitEthernet0/0
description Outside-LightPath
ip address 173.x.162 255.255.255.252
duplex auto
speed auto
!
interface GigabitEthernet0/1
description To Firewall
ip address 69.x.209 255.255.255.240
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
interface GigabitEthernet0/2
ip address 108.x.146 255.255.255.252
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat inside source list 50 interface GigabitEthernet0/2 overload
ip route 0.0.0.0 0.0.0.0 173.x.161
ip route 0.0.0.0 0.0.0.0 108.x.145
!
access-list 50 permit 69.x.211
!
no cdp run

!
!
control-plane
!
banner motd ^C
This equipment is privately owned & operated.
It is for AUTHORIZED USE ONLY!
All access to this equipment is logged.
DISCONNECT IMMEDIATELY if you are not an AUTHORIZED User.
Violators will be prosecuted to the fullest extent of the law.
^C
!
line con 0
login local
line aux 0
line vty 0 4
login local
transport input ssh
!
scheduler allocate 20000 1000
ntp server 128.118.25.5
end

[Philip D'Ath]

I think it should work, as cef will cache a (source,destination) pair and keep using the same outbound interface.  Otherwise if it did per packet load balancing things would be badly broken.

If you do a "show ip cef" it does show a whole lot of output, doesn't it?

The other rock solid option is to split the Internet in half, and route half the traffic down one circuit and the other down the other half.  Something like this would probably do the trick (untested):

ip route 0.0.0.0 128.0.0.0 173.x.161
ip route 128.0.0.0 128.0.0.0 108.x.145

[Darren Spezio]

I did a sh ip cef and there was quite a bit of output.

So if I add these 2 ip routes this will make it so that it goes out the same circuit it came in on?  

Also IP forwarding to internal servers is hit or miss for users which makes me packets are trying to go out a different circuit than which they came in.

i.e. Outside public address 69.x.x.212 to internal server 10.8.x.x - 

[Philip D'Ath]

You never mentioned inbound port forwarding.  That changes things a lot.

You are going to need to use policy routing.  Here is an example that directs outbound web browsing over the second NAT'd circuit.  You can change the access-list to divert whatever traffic you want.

ip access-list extended toGig02
 permit tcp any any eq www
 permit tcp any any eq 443

route-map toGig02 permit 10
 match ip address toGig02
 set ip next-hop 108.x.145

interface GigabitEthernet0/1
  ip policy route-map toGig02

[Darren Spezio]

So by simply making the circuits symmetrical will not make inbound port forwarding work properly?

[Philip D'Ath]

It would work for outbound requests, but not inbound requests to your severs,

Traffic to your servers could come in one circuit, and go out the other, which would break it.

[Darren Spezio]

Does this look correct? Should I remove "access-list 50 permit 69.x.208 0.0.0.15"?

Inet-2900#terminal length 0
! Last configuration change at 15:09:52 EDT Tue Apr 26 2016 
version 15.0

no ipv6 cef
ip source-route
ip cef

no ip domain lookup
ip domain name top.local
multilink bundle-name authenticated

interface GigabitEthernet0/0
description Outside-LightPath
ip address 173.x.162 255.255.255.252
duplex auto
speed auto
!
interface GigabitEthernet0/1
description To Firewall
ip address 69.x.209 255.255.255.240
ip nat inside
ip virtual-reassembly
ip policy route-map toGig02
duplex auto
speed auto
!
interface GigabitEthernet0/2
ip address 108.x.146 255.255.255.252
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat inside source list 50 interface GigabitEthernet0/2 overload
ip route 0.0.0.0 0.0.0.0 173.x.161
ip route 0.0.0.0 0.0.0.0 108.x.145
!
ip access-list extended toGig02
permit tcp any any eq www
permit tcp any any eq 443
!
access-list 50 permit 69.x.208 0.0.0.15
!
no cdp run

route-map toGig02 permit 10
match ip address toGig02
set ip next-hop 108.x.145

[Philip D'Ath]

Access-list 50 is unused in the configuration above, so yes you can remove it.

[Darren Spezio]

 I removed access-list 50 and lost all internet connections, I put it back and we're up again.

Inbound Server connections to 69.x.212 - 216 are still not working, what am I missing??

 Also, I noticed that our public facing IP address is 108.x.146 when before it was 69.x.211, I not sure if this is relevant, but I thought I would include it.

[Darren Spezio]

pinging...

[Philip D'Ath]

Sorry yes, access-list 50 is needed for the NAT configuration, so you do need to leave it in.

Yes, out outbound IP will now be different for http and https traffic.

So everything is working correctly now?

[Darren Spezio]

No it is not allowing inbound connections to our servers, what am I missing? 

Thanks for your help.

[Philip D'Ath]

Perhaps try adding:

route-map toGig02 permit 20

[Philip D'Ath · Accepted solution]

Remove this static route.  This is probably the issue.

ip route 0.0.0.0 0.0.0.0 108.x.145

You don't need a route via the provider that traffic is getting NATed.

[Darren Spezio]

Removal of the static route appears to have worked, I will monitor over the next few days and then give you the proper accolades.

Thank you Philip!