SSH to WAN with static nat won't work (dual ISP)

[Vadim Doroginin]

Hello all!

Everything worked fine untill... I introduced second ISP for redundancy.

I have C881 ISR installation with two ISP, both are NAT outside with failover.

When both ISP work fine, ISP1 is used to NAT internal users to outside internet and ISP2 is used to build DM_VPN Tunnel (Spoke).

Both ISPs are tracked by IP SLA.

When ISP1 goes down internal users are switched to NAT through ISP2.

When ISP2 goes down DMVPN Tunnel is switched to build through ISP1.

ISP1 - Fa4.41

ISP2 - Di0 through Fa4.42

Here is my problem: I can't SSH into ISP interfaces. Router actively refuses connection.

I narrowed down problem to following lines:

ip nat inside source static tcp 10.50.255.10 20502 isp1-ip.x.y.z 20502 extendable
ip nat inside source static udp 10.50.255.10 20502 isp1-ip.x.y.z 20502 extendable
ip nat inside source static tcp 10.50.255.10 3389 isp1-ip.x.y.z 23389 extendable
ip nat inside source static tcp 10.50.255.10 20502 isp2-ip.x.y.z 20502 extendable
ip nat inside source static udp 10.50.255.10 20502 isp2-ip.x.y.z 20502 extendable
ip nat inside source static tcp 10.50.255.10 3389 isp2-ip.x.y.z 23389 extendable

I have to remove these lines, then reload my router... then SSH starts working on ISP interfaces.

If I add these lines after my router is reloaded, SSH continues working until I reboot it.

So the problem is: if these lines are present at router startup, SSH won't work.

I have a workaround: make static nat isp:xxxxx - > 10.50.255.1:22, however I'm afraid that there is something else I'm missing in this installation which may cause more problems with more complex troubleshooting.

I've done my homework: there is pretty much people with the same problem and no solution (here is small part of them).

Here is what I've already tried:

• remove access-groups from external interfaces - won't help
• remove static NAT rules - won't help until reload
• remove NAT completely (all rules + nat inside/outside on interfaces) - yep, it works, however there is veryyyy small problem: I need NAT to access the Internet
• shutdown one of ISP - failover works great, however SSH still not working
• debug ip packet, routing, policy routing, nat: I get SYN from remote peer, then router responds with RST after FIBipv4-packet-proc: packet routing failed. Knowing this didn't help - why routing failed? packet comes with dst=ip-of-router... what is in it to fail?

Full config follows.

!
! Last configuration change at 22:51:41 GMT+4 Tue Oct 15 2013
version 15.3
no service pad
service timestamps debug datetime
service timestamps log datetime msec
service password-encryption
!
hostname archimed-gw
!
boot-start-marker
boot system flash:c880data-universalk9-mz.153-3.M.bin
boot-end-marker
!
!
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
memory-size iomem 10
clock timezone GMT+4 4 0
!
!
!
!
!
!
!
ip dhcp excluded-address 10.50.255.1 10.50.255.9
!
ip dhcp pool home_lan
 network 10.50.255.0 255.255.255.0
 domain-name xxx.loc
 dns-server 10.50.255.1
 default-router 10.50.255.1
 option 150 ip 10.177.20.1
!
ip dhcp pool Archimed-VAIO-WiFi
 host 10.50.255.10 255.255.255.0
 client-identifier 0108.edb9.ad80.2b
!
ip dhcp pool Archimed-VAIO-Eth
 host 10.50.255.15 255.255.255.0
 client-identifier 01f0.bf97.063a.80
!
ip dhcp pool Legova-Lenovo-WiFi
 host 10.50.255.12 255.255.255.0
 client-identifier 0100.16eb.2b0a.a4
!
!
!
ip domain name xxx.loc
ip cef
no ipv6 cef
!
!
multilink bundle-name authenticated
license udi pid CISCO881-PCI-K9 sn FCZ1715C1A6
!
!
!
!
!
!
track 1 ip sla 1 reachability
 delay down 60 up 60
!
track 2 ip sla 2 reachability
 delay down 60 up 60
!
ip ssh port 7522 rotary 1
ip ssh version 2
!
!
crypto isakmp policy 11
 encr aes 256
 authentication pre-share
 group 5
crypto isakmp key xxx address 0.0.0.0
!
!
crypto ipsec transform-set aes256-sha esp-aes 256 esp-sha-hmac
 mode transport
!
!
crypto ipsec profile dmvpn
 set transform-set aes256-sha
 set pfs group5
!
!
!
!
!
!
interface Tunnel10
 description DM_vpn
 ip address 192.168.254.10 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication xxx
 ip nhrp map multicast hub1.x.y.z
 ip nhrp map 192.168.254.254 hub1.x.y.z
 ip nhrp map 192.168.254.1 hub2.x.y.z
 ip nhrp map multicast hub2.x.y.z
 ip nhrp network-id 1
 ip nhrp nhs 192.168.254.1
 ip nhrp nhs 192.168.254.254
 ip nhrp registration no-unique
 ip virtual-reassembly in
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 0
 tunnel source Dialer0
 tunnel mode gre multipoint
 tunnel key 8
 tunnel route-via Dialer0 mandatory
 tunnel protection ipsec profile dmvpn
!
interface FastEthernet0
 no ip address
 shutdown
!
interface FastEthernet1
 no ip address
 shutdown
!
interface FastEthernet2
 no ip address
 shutdown
!
interface FastEthernet3
 switchport access vlan 10
 no ip address
!
interface FastEthernet4
 description Internet TRUNK
 no ip address
 ip access-group ext_if in
 duplex auto
 speed auto
!
interface FastEthernet4.41
 description Qwerty
 encapsulation dot1Q 41
 no ip dhcp client request domain-name
 no ip dhcp client request dns-nameserver
 ip address dhcp hostname QWERTY1
 ip access-group ext_if in
 no ip redirects
 ip nat outside
 ip virtual-reassembly in max-reassemblies 1024
!
interface FastEthernet4.42
 description Smile
 encapsulation dot1Q 42
 ip address dhcp
 ip access-group ext_if in
 no ip redirects
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
interface Vlan1
 no ip address
!
interface Vlan10
 description USER_vlan
 ip address 10.50.255.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip policy route-map LAN2WAN
!
interface Dialer0
 description Smile
 bandwidth 100000
 ip address negotiated
 ip access-group ext_if in
 no ip redirects
 ip mtu 1492
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 dialer pool 1
 ppp authentication chap callin
 ppp chap hostname xxx
 ppp chap password 7 yyyy
 ppp ipcp dns reject
 no cdp enable
!
router ospf 5
 router-id 10.50.255.1
 passive-interface Dialer0
 passive-interface FastEthernet4
 passive-interface FastEthernet4.41
 passive-interface FastEthernet4.42
 network 10.50.255.0 0.0.0.255 area 10.50.255.0
 network 192.168.254.0 0.0.0.255 area 0.0.0.0
!
ip local policy route-map Local
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip dns view xxx-local
 domain name xxx.loc
 domain name-server  10.177.14.50
 domain name-server  192.168.77.1
ip dns view default
 domain name-server  8.8.8.8
 domain name-server  8.8.4.4
ip dns view-list xxx-split
 view xxx-local 1
  restrict name-group 2
 view default 2
  restrict name-group 1
ip dns name-list 1 deny .*xxx.LOC
ip dns name-list 1 permit .*
ip dns name-list 2 permit .*xxx.LOC
ip dns name-list 2 deny .*
ip dns server view-group xxx-split
ip dns server
ip nat inside source route-map ISP1_NAT interface FastEthernet4.41 overload
ip nat inside source route-map ISP2_NAT interface Dialer0 overload
ip nat inside source static tcp 10.50.255.10 20502 isp1-ip.x.y.z 20502 extendable
ip nat inside source static udp 10.50.255.10 20502 isp1-ip.x.y.z 20502 extendable
ip nat inside source static tcp 10.50.255.10 3389 isp1-ip.x.y.z 23389 extendable
ip nat inside source static tcp 10.50.255.10 20502 isp2-ip.x.y.z 20502 extendable
ip nat inside source static udp 10.50.255.10 20502 isp2-ip.x.y.z 20502 extendable
ip nat inside source static tcp 10.50.255.10 3389 isp2-ip.x.y.z 23389 extendable
ip route 0.0.0.0 0.0.0.0 isp1-gw.x.y.z track 1
ip route 0.0.0.0 0.0.0.0 isp2-gw.x.y.z track 2
ip route 46.246.32.113 255.255.255.255 Dialer0 track 2
ip route 37.220.6.160 255.255.255.255 Dialer0 track 2
ip route 37.220.6.161 255.255.255.255 Dialer0 track 2
ip route 95.143.192.249 255.255.255.255 Dialer0 track 2
!
ip access-list standard ISP1_ROUTE
 permit isp1-ip.x.y.z
ip access-list standard ISP2_ROUTE
 permit isp2-ip.x.y.z
ip access-list standard NAT_ACL
 permit 10.50.255.0 0.0.0.255
!
ip access-list extended ext_if
 deny   tcp any any eq telnet
 deny   udp any any eq domain
 deny   tcp any any eq 22
 deny   tcp any any eq 3389
 deny   udp any any eq tftp
 permit ip any any
!
ip sla auto discovery
ip sla 1
 icmp-echo 194.87.0.50 source-interface FastEthernet4.41
 threshold 500
 timeout 3000
 frequency 3
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 194.87.0.50 source-interface Dialer0
 threshold 500
 timeout 3000
 frequency 3
ip sla schedule 2 life forever start-time now
logging history size 500
logging history debugging
logging trap debugging
logging host 10.50.255.10
!
route-map LAN2WAN permit 10
 match ip address NAT_ACL
 set default interface FastEthernet4.41
!
route-map Local permit 10
 match ip address ISP1_ROUTE
 set ip default next-hop isp1-gw.x.y.z
!
route-map Local permit 20
 match ip address ISP2_ROUTE
 set ip default next-hop isp2-gw.x.y.z
!
route-map ISP2_NAT permit 10
 match ip address NAT_ACL
 match interface Dialer0
!
route-map ISP1_NAT permit 10
 match ip address NAT_ACL
 match interface FastEthernet4.41
!
!
!
!
control-plane
!
!
!
line con 0
 privilege level 15
 logging synchronous
 no modem enable
line aux 0
line vty 0 4
 privilege level 15
 logging synchronous
 rotary 1
 transport input ssh
!
ntp server ip ru.pool.ntp.org
event manager applet DMVPNISPDown
 event track 2 state down
 action 10 cli command "enable"
 action 20 cli command "configure terminal"
 action 30 cli command "interface Tunnel10"
 action 40 cli command "shutdown"
 action 50 cli command "tunnel source FastEthernet4.41"
 action 60 cli command "tunnel route-via FastEthernet4.41 mandatory"
 action 70 cli command "no shutdown"
 action 90 cli command "end"
event manager applet DMVPNISPUp
 event track 2 state up
 action 10 cli command "enable"
 action 20 cli command "configure terminal"
 action 30 cli command "interface Tunnel10"
 action 40 cli command "shutdown"
 action 50 cli command "tunnel source Dialer0"
 action 60 cli command "tunnel route-via Dialer0 mandatory"
 action 70 cli command "no shutdown"
 action 90 cli command "end"
event manager applet NATISPDown
 event track 1 state down
 action 10 cli command "enable"
 action 20 cli command "configure terminal"
 action 30 cli command "route-map LAN2WAN permit 10"
 action 40 cli command "no set default interface Dialer0"
 action 41 cli command "no set default interface FastEthernet4.41"
 action 50 cli command "set default interface Dialer0"
 action 60 cli command "end"
 action 70 cli command "clear ip nat translation *"
 action 80 cli command "clear ip nat translation forced"
event manager applet NATISPUp
 event track 1 state up
 action 10 cli command "enable"
 action 20 cli command "configure terminal"
 action 30 cli command "route-map LAN2WAN permit 10"
 action 40 cli command "no set default interface Dialer0"
 action 41 cli command "no set default interface FastEthernet4.41"
 action 50 cli command "set default interface FastEthernet4.41"
 action 60 cli command "end"
 action 70 cli command "clear ip nat translation *"
 action 80 cli command "clear ip nat translation forced"
!
end

Terminal output when remote peer tries to connect to router:7522

Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly(36), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Access List(42), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly After IPSec Decryption(52), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, NAT Outside(80), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: FIBipv4-packet-proc: route packet from FastEthernet4.41 src remote-ip.x.y.z dst isp1-ip.x.y.z
Oct 15 17:53:04: FIBfwd-proc: Default:isp1-ip.x.y.z/32 receive entry
Oct 15 17:53:04: FIBipv4-packet-proc: packet routing failed
Oct 15 17:53:04: IP: tableid=0, s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), routed via RIB
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, rcvd 3
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, stop process pak for forus packet
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, enqueue feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, local feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, policy match
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: route map Local, item 10, permit
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, policy routed
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: local to FastEthernet4.41 isp1-gw.x.y.z
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, local feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Policy Routing(3), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Post-routing NAT Outside(24), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Common Flow Table(27), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Stateful Inspection(28), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
archimed-gw#
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT ALG proxy(59), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending full packet
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: SSH2 0: channel window adjust message received 9492
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly(36), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Access List(42), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly After IPSec Decryption(52), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, NAT Outside(80), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, input feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: FIBipv4-packet-proc: route packet from FastEthernet4.41 src remote-ip.x.y.z dst isp1-ip.x.y.z
Oct 15 17:53:04: FIBfwd-proc: Default:isp1-ip.x.y.z/32 receive entry
Oct 15 17:53:04: FIBipv4-packet-proc: packet routing failed
Oct 15 17:53:04: IP: tableid=0, s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), routed via RIB
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, output feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 52, rcvd 3
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, stop process pak for forus packet
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:04: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 52, enqueue feature
Oct 15 17:53:04:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, local feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, policy match
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: route map Local, item 10, permit
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, policy routed
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: local to FastEthernet4.41 isp1-gw.x.y.z
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, local feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Policy Routing(3), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Post-routing NAT Outside(24), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Common Flow Table(27), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Stateful Inspection(28), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT ALG proxy(59), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:04: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending full packet
Oct 15 17:53:04:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly(36), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Access List(42), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Virtual Fragment Reassembly After IPSec Decryption(52), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, NAT Outside(80), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, input feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: FIBipv4-packet-proc: route packet from FastEthernet4.41 src remote-ip.x.y.z dst isp1-ip.x.y.z
Oct 15 17:53:05: FIBfwd-proc: Default:isp1-ip.x.y.z/32 receive entry
Oct 15 17:53:05: FIBipv4-packet-proc: packet routing failed
Oct 15 17:53:05: IP: tableid=0, s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), routed via RIB
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 48, output feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 48, output feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Common Flow Table(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 48, output feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, Stateful Inspection(28), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z (FastEthernet4.41), len 48, rcvd 3
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, stop process pak for forus packet
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN
Oct 15 17:53:05: IP: s=remote-ip.x.y.z (FastEthernet4.41), d=isp1-ip.x.y.z, len 48, enqueue feature
Oct 15 17:53:05:     TCP src=8380, dst=7522, seq=999405847, ack=0, win=8192 SYN, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, local feature
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z, len 40, policy match
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:05: IP: route map Local, item 10, permit
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, policy routed
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:05: IP: local to FastEthernet4.41 isp1-gw.x.y.z
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, local feature
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Policy Routing(3), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Post-routing NAT Outside(24), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Common Flow Table(27), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, Stateful Inspection(28), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, output feature
archimed-gw#
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST, NAT ALG proxy(59), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 15 17:53:05: IP: s=isp1-ip.x.y.z (local), d=remote-ip.x.y.z (FastEthernet4.41), len 40, sending full packet
Oct 15 17:53:05:     TCP src=7522, dst=8380, seq=0, ack=999405848, win=0 ACK RST
Oct 15 17:53:05: SSH2 0: channel window adjust message received 8206

Please, help!

[Edgardo Rodriguez]

I´m stuck with the same annoying issue. It´s incredible that no one came across with this, the info you provided here couldn't be more complete.
Have you ever found any solution?

[Vadim Doroginin]

Nope, I had to remove one of translations. I can live with that because it's not critical for me.

If it is critical there is two solutions:

1. Use BGP - it will solve all your problems :).

2. Use one static NAT. When ISP is down rewrite NAT rules inside EEM applet.

[Edgardo Rodriguez]

Hi!

By BGP, item #1, you mean inter-VRF routing using this protocol?

This magically solves these issues? If so i´ll try it...

[Vadim Doroginin]

Nope, I mean actual BGP with your own AS+subnet and BGP-peering with both ISPs.

Long story short: some internal resource (say http server 192.168.1.123:80) can not be published to internet with dual active IP addresses from your ISPs.

• You can easily publish one resource to one IP (say x.x.x.x:80=192.168.1.123:80) and another resource to second IP (say y.y.y.y:443=192.168.1.123:443).
• You can failover resource from first IP to second IP and back to the first IP at your wish or by means of EEM applets.
• You can publish a resource on an alternative IP for some specific subnet (say y.y.y.y:80=192.168.1.123:80 for z.z.z.0/24 and x.x.x.x:80=192.168.1.123:80 for the rest of internet).

There is simple explanation for this limitation: even when incoming packet is correctly routed, NATed and firewalled from some public IP to one of your public IPs (say w.w.w.w:9999 is connecting to secondary IP y.y.y.y:80 which correctly translates to w.w.w.w:9999->192.168.1.123:80), the outgoing packet will always have a specific route through a specific IP (say a response is sent from 192.168.1.123:80 to w.w.w.w:9999 which translates to x.x.x.x:80->w.w.w.w:9999). You can see why it won't work because w.w.w.w expectes a response from y.y.y.y, not from x.x.x.x.

So go ahead and remove duplicate static-NAT lines from your config.

[Edgardo Rodriguez]

Ok thanks for your detailed explanation.

My solution was making a SNAT for one of the static-nat because in my scenario there is no way to publish this service on a different IP Address. Example: 192.168.20.5:80 has to be accessible from internet, any public source address and translated x.x.x.x:5955->192.168.20.5:80 AND y.y.y.y:5955->192.168.20.5:80, both Active simultaneously.

Regards.

[Philip D'Ath]

I've seen this problem before.  It was an IOS bug.  From memory, we hit it in 15.4(3)Mx train.  I think it was in some others as well.

We resolved the issue by upgrading to a newer version of IOS.

[Edgardo Rodriguez]

Yes! I am aware of that, I successfully configured routers running newer IOS without this issue, but we have a few 18xx that can not be pushed beyond 15.1.x version

[Philip D'Ath]

Consider downgrading then until the issue goes away.

[Edgardo Rodriguez]

Just for the record..
Unfortunately the "newer" version that works exactly with the configuration layout we are using is 15.0M.
It´s pretty old and we have had two important issues:
1- Auth by radius server fails: Passwords sent to the server are messed up (confirmed by radius debugging)
2- Stability is bad: Two router hangs in one week.
So we can not consider this an option.
BTW thanks for your recommendation.