VPN return traffic not flowing over the tunnel

[bautsche123]

Hello.

I've tried to find something on the internet to solve this, but am failing miserably. I guess, I really don't understand how the cisco decides on routing.

Anyway, I have a Cisco 837 which I'm using for internet access and which I would like to be able to terminate a VPN on. When I vpn in (using vpnc from a Solaris box as it happens which is connected to the ethernet interface of the cisco), I can establish an VPN and when I ping a host on the inside, I can see that ping packet arrive, however the return packet, the cisco 837 tries to send over the public internet facing interface Dialer1 without encryption. I can't for the life of me work out why.

(Also note: I can also establish a tunnel from the public internet, but again, I can't get any traffic back through the tunnel. I assume that I'm having the same issue, ie return packets aren't going where they should, but I don't know that for certain, on the host being pinged though, I can see the ping packets arriving and the host responding with an ICMP Echo reply).

here is the cisco version:

adsl#show version
Cisco IOS Software, C850 Software (C850-ADVSECURITYK9-M), Version 12.4(15)T5, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Thu 01-May-08 02:07 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)YI4, RELEASE SOFTWARE

adsl uptime is 1 day, 19 hours, 27 minutes
System returned to ROM by power-on
System restarted at 17:20:56 bst Sun Oct 10 2010
System image file is "flash:c850-advsecurityk9-mz.124-15.T5.bin"

Cisco 857 (MPC8272) processor (revision 0x300) with 59392K/6144K bytes of memory.
Processor board ID FCZ122391F5
MPC8272 CPU Rev: Part Number 0xC, Mask Number 0x10
4 FastEthernet interfaces
1 ATM interface
128K bytes of non-volatile configuration memory.
20480K bytes of processor board System flash (Intel Strataflash)

Configuration register is 0x2102

And here is the cisco's configuration (IP address, etc changed of course):

Current configuration : 7782 bytes
!
! Last configuration change at 11:57:21 bst Mon Oct 11 2010 by bautsche
! NVRAM config last updated at 11:57:22 bst Mon Oct 11 2010 by bautsche
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname adsl
!
boot-start-marker
boot-end-marker
!
logging buffered 4096
enable secret 5 <secret>
!
aaa new-model
!
!
aaa authentication login local_authen local
aaa authentication login sdm_vpn_xauth_ml_1 local
aaa authorization exec local_author local
aaa authorization network sdm_vpn_group_ml_1 local
!
!
aaa session-id common
clock timezone gmt 0
clock summer-time bst recurring last Sun Mar 1:00 last Sun Oct 1:00
!
!
dot11 syslog
no ip source-route
ip dhcp database dhcpinternal
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.7.1 10.10.7.99
ip dhcp excluded-address 10.10.7.151 10.10.7.255
!
ip dhcp pool dhcpinternal
   import all
   network 10.10.7.0 255.255.255.0
   default-router 10.10.7.1
   dns-server 212.159.6.9 212.159.6.10 212.159.13.49 212.159.13.50
!
!
ip cef
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
no ip bootp server
ip host nfs1 10.10.140.207
ip name-server 212.159.11.150
ip name-server 212.159.13.150
!
!
!
username cable password 7 <password>
username bautsche password 7 <password>
username vpnuser password 7 <password>
!
!
crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2
!
crypto isakmp policy 2
encr aes 256
authentication pre-share
group 2
!
crypto isakmp policy 3
encr 3des
authentication pre-share group 2
crypto isakmp client configuration address-pool local SDM_POOL_1
!
crypto isakmp client configuration group groupname2
key <key>
dns 10.10.140.201 10.10.140.202
domain swangage.co.uk
pool SDM_POOL_1
max-users 3
netmask 255.255.255.0
!
crypto isakmp client configuration group groupname1
key <key>
dns 10.10.140.201 10.10.140.202
domain swangage.co.uk
pool SDM_POOL_1
max-users 3
netmask 255.255.255.0
crypto isakmp profile sdm-ike-profile-1
   match identity group groupname2
   client authentication list sdm_vpn_xauth_ml_1
   isakmp authorization list sdm_vpn_group_ml_1
   client configuration address respond
crypto isakmp profile sdm-ike-profile-2
   match identity group groupname1
   isakmp authorization list sdm_vpn_group_ml_1
   client configuration address respond
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP_MD5_3DES esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes esp-sha-hmac
!
crypto dynamic-map SDM_DYNMAP_1 1
set security-association idle-time 3600
set transform-set ESP-AES-256-SHA
reverse-route
crypto dynamic-map SDM_DYNMAP_1 2
set security-association idle-time 3600
set transform-set ESP-AES-256-SHA
reverse-route
!
!
crypto map SDM_CMAP_1 client authentication list sdm_vpn_xauth_ml_1
crypto map SDM_CMAP_1 isakmp authorization list sdm_vpn_group_ml_1
crypto map SDM_CMAP_1 65535 ipsec-isakmp dynamic SDM_DYNMAP_1
!
crypto ctcp port 10000
archive
log config
  hidekeys
!
!
ip tcp synwait-time 10
!
!
!
interface Null0
no ip unreachables
!
interface ATM0
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
no atm ilmi-keepalive
pvc 0/38
  encapsulation aal5mux ppp dialer
  dialer pool-member 1
!
dsl operating-mode auto
hold-queue 224 in
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface Vlan1
description $FW_INSIDE$
ip address 10.10.7.1 255.255.255.0
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip route-cache flow
crypto map SDM_CMAP_1
hold-queue 100 out
!
interface Dialer1
description $FW_OUTSIDE$
ip address negotiated
ip access-group 121 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
encapsulation ppp
ip route-cache flow
no ip split-horizon
dialer pool 1
dialer idle-timeout 0
dialer persistent
dialer-group 1
no cdp enable
ppp authentication chap callin
ppp chap hostname <hostname>
ppp chap password 7 <password>
crypto map SDM_CMAP_1
!
ip local pool SDM_POOL_1 10.10.148.11 10.10.148.20
ip local pool public_184 123.12.12.184
ip local pool public_186 123.12.12.186
ip local pool public_187 123.12.12.187
ip local pool internal_9 10.10.7.9
ip local pool internal_8 10.10.7.8
ip local pool internal_223 10.10.7.223
ip local pool internal_47 10.10.7.47
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 10.10.140.0 255.255.255.0 10.10.7.2
!
no ip http server
no ip http secure-server
ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload
ip nat inside source static 10.10.7.9 123.12.12.184
ip nat inside source static tcp 10.10.7.8 22 123.12.12.185 22 extendable
ip nat inside source static tcp 10.10.7.8 25 123.12.12.185 25 extendable
ip nat inside source static tcp 10.10.7.8 80 123.12.12.185 80 extendable
ip nat inside source static tcp 10.10.7.8 443 123.12.12.185 443 extendable
ip nat inside source static tcp 10.10.7.8 993 123.12.12.185 993 extendable
ip nat inside source static tcp 10.10.7.8 1587 123.12.12.185 1587 extendable
ip nat inside source static tcp 10.10.7.8 8443 123.12.12.185 8443 extendable
ip nat inside source static 10.10.7.223 123.12.12.186
ip nat inside source static 10.10.7.47 123.12.12.187
!
logging 10.10.140.213
access-list 18 permit any
access-list 23 permit 10.10.140.0 0.0.0.255
access-list 23 permit 10.10.7.0 0.0.0.255
access-list 100 remark SDM_ACL Category=2
access-list 100 deny   ip any 10.10.148.0 0.0.0.255
access-list 100 permit ip any any
access-list 121 remark SDM_ACL Category=17
access-list 121 deny   udp any eq netbios-dgm any
access-list 121 deny   udp any eq netbios-ns any
access-list 121 deny   udp any eq netbios-ss any
access-list 121 deny   tcp any eq 137 any
access-list 121 deny   tcp any eq 138 any
access-list 121 deny   tcp any eq 139 any
access-list 121 permit ip any any
access-list 125 permit tcp any any eq www
access-list 125 permit udp any eq isakmp any
access-list 125 permit udp any any eq isakmp
access-list 194 deny   udp any eq isakmp any
access-list 194 deny   udp any any eq isakmp
access-list 194 permit ip host 123.12.12.184 any
access-list 194 permit ip any host 123.12.12.184
access-list 194 permit ip host 10.10.7.9 any
access-list 194 permit ip any host 10.10.7.9
access-list 195 deny   udp any eq isakmp any
access-list 195 deny   udp any any eq isakmp
access-list 195 permit ip host 123.12.12.185 any
access-list 195 permit ip any host 123.12.12.185
access-list 195 permit ip host 10.10.7.8 any
access-list 195 permit ip any host 10.10.7.8
no cdp run
route-map public_185 permit 10
match ip address 195
!
route-map public_184 permit 10
match ip address 194
!
route-map SDM_RMAP_1 permit 1
match ip address 100
!
!
control-plane
!
!
line con 0
login authentication local_authen
no modem enable
transport preferred none
transport output telnet
stopbits 1
line aux 0
login authentication local_authen
transport output telnet
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
authorization exec local_author
login authentication local_authen
length 0
transport preferred none
transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
sntp server 130.88.202.49
sntp server 130.88.200.98
sntp server 130.88.200.6
sntp server 130.88.203.64
end

Any help would be appreciated.

Thanks a lot.

Ciao,

Eric

[bautsche123]

Hi Herbert.

I'll redo all the debugging and snoops, it'll probably be wednesday though.

One thing I can say for certain though is that I definitely didn't get the masking of addresses wrong in the previous post. The interal address of the host did not turn up in the NAT debug output, it was the two external ISP addresses that I have (one on the client side, one on the VPN server side).

Thanks again for your kind help.

Eric

[Herbert Baerten]

Ok, in the meantime some more questions

What is 10.10.40.8 ? Is it the host running vpnc or something else?

I'm asking because I spotted this:

ip nat inside source list 18 interface Dialer1 overload
ip nat inside source static 10.10.40.8 interface Dialer1

You're using the same public ip for both dynamic nat (overload) and static nat towards one host ?

Not sure if this can be expected to work..

BTW I would also like to have another look at what is working versus what is not working... what is different? the working connection to the company vpn is it from the same vpnc client? or from the hardware client you mentioned?

[bautsche123]

Hi Herbert.

10.10.40.8 is my firewall connected to that ISP. It's handling incoming traffic, e.g. ssh, web, etc and dropping everything else it gets.

Re the vpn to the company, I can vpn in using the same vpnc client on the same laptop (with a different configuration of course).

Thanks again for your help.

Ciao,

Eric

[bautsche123]

Sorry for the delay.

I've removed the static nat line

ip nat inside source static 192.168.40.8 interface Dialer1

and I now just have

ip nat inside source list 18 interface Dialer1 overload

in the configuration of the router on the client side.

Here is the snoop output from the client while trying to connect:

10.10.40.106 -> 123.1.1.185 UDP D=500 S=500 LEN=1294
123.1.1.185 -> 10.10.40.106 UDP D=500 S=500 LEN=412
10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168
10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168
10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168
10.10.40.106 -> 123.1.1.185 UDP D=4500 S=4500 LEN=168

And here is the debug output on the router on the client side:

1w6d: NAT*: s=10.10.40.106->122.1.1.142, d=123.1.1.185 [7823]
1w6d: NAT*: o: udp (123.1.1.185, 500) -> (122.1.1.142, 500) [20120]
1w6d: NAT*: s=123.1.1.185, d=122.1.1.142->10.10.40.106 [20120]
1w6d: NAT*: i: udp (10.10.40.106, 4500) -> (123.1.1.185, 4500) [7824]
1w6d: NAT*: s=10.10.40.106->122.1.1.142, d=123.1.1.185 [7824]
1w6d: NAT*: i: udp (10.10.40.106, 4500) -> (123.1.1.185, 4500) [7825]
1w6d: NAT*: s=10.10.40.106->122.1.1.142, d=123.1.1.185 [7825]
1w6d: IP: s=123.1.1.185 (Dialer1), d=122.1.1.142 (Dialer1), len 124, rcvd 3
1w6d:     UDP src=1029, dst=4500
1w6d: IP: s=123.1.1.185 (Dialer1), d=122.1.1.142 (Dialer1), len 116, rcvd 3
1w6d:     UDP src=1029, dst=4500
1w6d: IP: s=123.1.1.185 (Dialer1), d=122.1.1.142 (Dialer1), len 100, rcvd 3
1w6d:     UDP src=1029, dst=4500
1w6d: %CRYPTO-4-IKMP_NO_SA: IKE message from 123.1.1.185   has no SA and is not an initialization offer
1w6d: NAT: Allocated Port for 10.10.40.106 -> 122.1.1.142: wanted 47198 got 47198
1w6d: NAT: Allocated Port for 10.10.40.106 -> 122.1.1.142: wanted 36959 got 36959
1w6d: NAT*: i: udp (10.10.40.106, 4500) -> (123.1.1.185, 4500) [7828]
1w6d: NAT*: s=10.10.40.106->122.1.1.142, d=123.1.1.185 [7828]
1w6d: NAT: Allocated Port for 10.10.40.106 -> 122.1.1.142: wanted 47928 got 47928
1w6d: NAT*: i: udp (10.10.40.106, 4500) -> (123.1.1.185, 4500) [7829]
1w6d: NAT*: s=10.10.40.106->122.1.1.142, d=123.1.1.185 [7829]

Finally, here is the show ip nat trans

udp 122.1.1.142:4500 10.10.40.106:4500 123.1.1.185:4500 123.1.1.185:4500
udp 122.1.1.142:500 10.10.40.106:500 123.1.1.185:500  123.1.1.185:500

So we now have the ports issue sorted but we still don't get the response packet on port 4500 to cross back to the client.... :-(

Eric

[Herbert Baerten]

Hi Eric,

there are 2 things I found interesting here:

1w6d: IP: s=123.1.1.185 (Dialer1), d=122.1.1.142 (Dialer1), len 100, rcvd 3
1w6d:     UDP src=1029, dst=4500

so 123.1.1.185 (that is your VPN router, right?) is sending the return packet with source port 1029 - that does not make any sense... or maybe something in the path is modifying the port?

Now given that it has the wrong source port, it is logical that it gets dropped - it does not match the entry in the NAT table.

Did you get the (crypto isakmp) debugs on the vpn router as well?

If not can you please repeat the test and get debug crypto isakmp on the vpnrouter, as well as the same debugs as before (nat, packet) on the nat router at the same time.

Second thing:

1w6d: %CRYPTO-4-IKMP_NO_SA: IKE message from 123.1.1.185   has no SA and is not an initialization offer

This indicates that crypto is enabled on this router, and that it considers the UDP4500 packet as destined for itself. Looking at the config you posted earlier though there is no crypto enabled on this router... or did you change that in the meantime?

Herbert

[bautsche123]

I suspect the VPN router is sending the packet out with port 1029 because of the NAT configured on it:

ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload
ip nat inside source static tcp 10.10.7.8 22 123.1.1.185 22 extendable
ip nat inside source static tcp 10.10.7.8 25 123.1.1.185 25 extendable
ip nat inside source static tcp 10.10.7.8 80 123.1.1.185 80 extendable
ip nat inside source static tcp 10.10.7.8 443 123.1.1.185 443 extendable
ip nat inside source static tcp 10.10.7.8 993 123.1.1.185 993 extendable
ip nat inside source static tcp 10.10.7.8 1587 123.1.1.185 1587 extendable
ip nat inside source static tcp 10.10.7.8 8443 123.1.1.185 8443 extendable

The IP on Dialer1 is 123.1.1.185.

I just tried to replace the overload line with

ip nat inside source route-map SDM_RMAP_1 pool public_187 overload

Where public_187 is defined as:

ip local pool public_187 123.1.1.187

But this appears to stop all outgoing traffic. Any great ideas how to configure this properly?

Re the crypto message, no I haven't got crypto turned on, but does that not say that it's receiving a crypto packet rather than sending one?

Eric

[bautsche123]

OK, I now know why this didn't work:

the pool in this command

ip nat inside source route-map SDM_RMAP_1 pool public_187 overload

needs to be an ip nat pool.

An IP nat pool has to have a netmask of 30 or less and I haven't got enough IP addresses for that, so I can't do the overload using a pool.

The only other option instead of "pool" is "interface" and that leaves me with my .185 address as that's the address on the interface and it gets assigned automatically.

I wonder if there is a way to have the VPN server listen on an IP that's not on an interface?

Eric

[Herbert Baerten · Accepted solution]

Hi Eric,

(sorry for the delayed response - was in need of some vacation )

So I see you got a few steps farther now. I think there are 2 things we can try:

1)

I suppose you have put back this:

ip nat inside source route-map SDM_RMAP_1 interface Dialer1 overload

Since the routemap refers to ACL 100 to define the traffic to be translated, we can exclude the traffic that initiates from the router:

access-list 100 remark SDM_ACL Category=2

access-list 100 deny ip host 123.12.12.185 any
access-list 100 deny   ip any 10.10.148.0 0.0.0.255
access-list 100 permit ip any any

That should prevent the udp source port from changing from 4500 to 1029

OR

2)

if you prefer to use another ip address for VPN,

then you can use a loopback like this:

interface loopback 0

  ip address 123.12.12.187 255.255.255.255

  no shut

crypto map SDM_CMAP_1 local-address loopback 0

I don't think you need to apply the crypto map to the loopback interface, but it's been a while since I configured something like this, so if you have any trouble try that first, and if still not working get the crypto debugs again (isakmp+ipsec on the vpn router, nat+packet on the client router).

hth

Herbert

[bautsche123]

You are my star. Thank you very much - it now works.

I tried the option with the loopback interface first, unfortunately to no effect. I tried applying the crypto map to the loop back, too, again, not working.

I then tried the other option - changing the ACL to exclude the interface that the VPN server listens on.

Everything works now. I'm thoroughly impressed with your understanding of Cisco products.

I'm a very happy man now.

I dare say, now that I have the VPN working, I could also move it to the other IP, but there really is no need.

Thanks again for your perseverance and all your help in getting this sort. It's very very much appreciated.

Eric

[bautsche123]

Oh, I should also say that vpnc is working with force-natt

Eric

[Herbert Baerten]

Excellent, and believe me, I'm a very happy man as well now (and I agree, if it works now don't touch it any more )

cheers

Herbert