Cisco ISR with site to site VPN. Tunnel is up but traffic will not pass.

[wseyller]

Using Cisco ISR 1841

I can see some traffic from the IPSec VPN on the wan interface when the other side tries to ping to printers on the local lan.  There is no traffic from the VPN on the lan side.  The tunnel shows to be up on both sides.  The other side is using a fortigate firewall in a datacenter.

Here is my configuration.  The server at 10.1.2.57/32 is unable to ping a printer or anything else for example at 192.168.55.250.

I am not listing any acl for the wan interface because I removed it anyway during this testing.  Also I am list fake public ip addresses to censor.

version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable secret 5 ************************
!
no aaa new-model
!
resource policy
!
clock timezone EST -5
clock summer-time EST recurring 2 Sun Mar 2:00 1 Sun Nov 2:00
ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.55.1 192.168.55.99
ip dhcp excluded-address 192.168.55.150 192.168.55.254
!
ip dhcp pool TASK55
network 192.168.55.0 255.255.255.0
default-router 192.168.55.1
domain-name somedomain.int
dns-server 192.168.1.10 192.168.55.1
!
!
no ip domain lookup
ip domain name somedomain.int
ip name-server 192.168.1.10
!
!
crypto pki trustpoint TP-self-signed-25944030
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-25944030
revocation-check none
rsakeypair TP-self-signed-25944030
!
!
crypto pki certificate chain TP-self-signed-25944030
certificate self-signed 01
**************************************
quit
username ais privilege 15 password 7 ******************
!
!
crypto isakmp policy 2
encr 3des
authentication pre-share
group 2
crypto isakmp key SecretPass address 2.2.2.2
crypto isakmp keepalive 3600
crypto isakmp aggressive-mode disable
!
crypto ipsec security-association lifetime seconds 28800
!
crypto ipsec transform-set TASK_TS esp-3des esp-sha-hmac
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
description Tunnel to2.2.2.2
set peer 2.2.2.2
set transform-set TASK_TS
set pfs group2
match address 100
!
!
!
!
interface Tunnel0  (NOT RELATED TO IPSEC TUNNEL)
ip address 10.0.0.6 255.255.255.252
ip mtu 1476
tunnel source 1.1.1.1
tunnel destination 10.10.10.10
!
interface FastEthernet0/0
ip address 1.1.1.1 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
no mop enabled
crypto map SDM_CMAP_1
!
interface FastEthernet0/1
ip address 192.168.55.1 255.255.255.0
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
router ospf 1
router-id 192.168.55.1
log-adjacency-changes
network 192.168.55.0 0.0.0.255 area 0

network 10.0.0.4 0.0.0.3 area 0
!
!
!
ip http server
ip http authentication local
ip http secure-server
ip nat inside source list NATLIST interface FastEthernet0/0 overload
!
ip access-list extended NATLIST
deny ip 192.168.55.0 0.0.0.255 host 10.1.2.57
permit ip 192.168.55.0 0.0.0.255 any
!
access-list 100 remark CCP_ACL Category=4
access-list 100 remark IPSec Rule
access-list 100 permit ip 192.168.55.0 0.0.0.255 host 10.1.2.57
!
!
!
!
!
!
control-plane
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
exec-timeout 0 0
privilege level 15
logging synchronous
login local
transport input ssh
line vty 5 15
exec-timeout 0 0
logging synchronous
login local
transport input ssh
!
scheduler allocate 20000 1000
ntp clock-period 17179132
end

[Georg Pauwen]

Hello,

try and disable volume rekeying and replay detection in the crypto map:

set security-association lifetime kilobytes disable
set security-association replay disable

[wseyller]

the lifetime kilobytes disable command was not an option on this router.  the  disable command was not present only could set a value of 2560-536870912.

I did add the replay disable command.  I save the changes.  I decided to execute a clear crypto isakmp and clear crypto sa on the router and also reloaded the router.

Tried another ping with the debug ip packet running.

CTD-TASK-RTR#ping 10.1.2.57 source f0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.2.57, timeout is 2 seconds:
Packet sent with a source address of 192.168.55.1

*Oct 7 22:57:22.863: IP: tableid=0, s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:22.863: IP: s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), len 100, sending.
*Oct 7 22:57:24.863: IP: tableid=0, s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:24.863: IP: s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), len 100, sending.
*Oct 7 22:57:26.863: IP: tableid=0, s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:26.863: IP: s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), len 100, sending.
*Oct 7 22:57:28.863: IP: tableid=0, s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:28.863: IP: s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), len 100, sending.
*Oct 7 22:57:30.863: IP: tableid=0, s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:30.863: IP: s=192.168.55.1 (local), d=10.1.2.57 (FastEthernet0/0), len 100, sending.
Success rate is 0 percent (0/5)
CTD-TASK-RTR#
CTD-TASK-RTR#
CTD-TASK-RTR#
*Oct 7 22:57:34.239: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), routed via RIB
*Oct 7 22:57:34.239: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), g=192.168.55.251, len 60, forward
*Oct 7 22:57:34.239: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), len 60, encapsulation failed
CTD-TASK-RTR#
*Oct 7 22:57:42.475: IP: tableid=0, s=2.2.2.2 (FastEthernet0/0), d=1.1.1.1 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:42.475: IP: s=2.2.2.2 (FastEthernet0/0), d=1.1.1.1 (FastEthernet0/0), len 120, rcvd 3
*Oct 7 22:57:42.479: IP: tableid=0, s=1.1.1.1 (local), d=2.2.2.2 (FastEthernet0/0), routed via RIB
*Oct 7 22:57:42.479: IP: s=1.1.1.1 (local), d=2.2.2.2 (FastEthernet0/0), len 120, sending
CTD-TASK-RTR#show crypto
CTD-TASK-RTR#show crypto isa
CTD-TASK-RTR#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
1.1.1.1 2.2.2.2 QM_IDLE 1001 0 ACTIVE

IPv6 Crypto ISAKMP SA

[Georg Pauwen]

Hello,

looking at your original post, it looks like you are running IOS version 12.4, which is quite old. Can you post the output of 'show ver' ?

[Richard Burts]

There are some very interesting messages in the output that you posted. I include the first two to provide context. The really interesting message is the third one

*Oct 7 22:57:34.239: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), routed via RIB
*Oct 7 22:57:34.239: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), g=192.168.55.251, len 60, forward
*Oct 7 22:57:34.239: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.251 (FastEthernet0/1), len 60, encapsulation failed

Encapsulation failed on an Ethernet interface usually indicates some problem with arp. Would you post the output of the command show arp

HTH

Rick

[wseyller]

CTD-TASK-RTR#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.55.119 0 f09f.c220.cd9e ARPA FastEthernet0/1
Internet 192.168.55.118 45 d481.d7aa.a06a ARPA FastEthernet0/1
Internet 192.168.55.115 1 b083.fea5.e85f ARPA FastEthernet0/1
Internet 192.168.55.114 44 b083.fea5.e2aa ARPA FastEthernet0/1
Internet 192.168.55.113 44 f8bc.1261.357e ARPA FastEthernet0/1
Internet 192.168.55.112 1 f862.14aa.3228 ARPA FastEthernet0/1
Internet 192.168.55.1 - 0017.9578.13f1 ARPA FastEthernet0/1
Internet 1.1.1.1 - 0017.9578.13f0 ARPA FastEthernet0/0
Internet 192.168.55.39 1 f8b1.56b2.c388 ARPA FastEthernet0/1
Internet 192.168.5.254 0 1c1b.6828.f730 ARPA FastEthernet0/0
Internet 192.168.55.251 0 0021.b77b.ff10 ARPA FastEthernet0/1
Internet 192.168.55.250 0 0021.b7fb.5213 ARPA FastEthernet0/1
Internet IP of ISPs GATEWAY 45 1c1b.6828.f730 ARPA FastEthernet0/0
CTD-TASK-RTR#

[wseyller]

Cisco IOS Software, 1841 Software (C1841-ADVENTERPRISEK9-M), Version 12.4(9)T1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Wed 30-Aug-06 14:54 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)T9, RELEASE SOFTWARE (fc1)

CTD-TASK-RTR uptime is 12 hours, 59 minutes
System returned to ROM by reload at 18:54:23 EST Sun Oct 7 2018
System image file is "flash:c1841-adventerprisek9-mz.124-9.t1.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1841 (revision 6.0) with 118784K/12288K bytes of memory.
Processor board ID FTX1025Y06A
2 FastEthernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
500472K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

[Georg Pauwen]

Hello,

you are running very old software on a very old router, which I suspect might be causing your problems. Unfortunately your router does not have enough DRAM/Flash to run a higher (>15) version, so you cannot test if upgrading would solve your problem.

Is this a business environment ? Is there a chance to acquire a newer model router ?

[wseyller]

Back in the office.  We were discussing our options.  Is there any chance this could be an issue within the ISP equipment.

Otherwise we are looking at purchasing a Fortigate 30e firewall .

[Georg Pauwen]

As stated, you are running software and hardware that is older than 10 years. Try first to have your ISP configure the Fortigate with the lowest encryption possible (which is group 1). 

Also, have your ISP do and end to end test to your equipment. 

[wseyller]

Thanks for the info.  I had the ISP check and compare their modem with all the other sites and they confirm everything was the same.  I am currently talking with the ISP currently before we purchase the Fortigate.  I will have to perform the configuration of the Fortigate as it will be the customers equipment.  I will give a thought to use DH group 1.  Originally that is what the vendor wanted to use but the Ubiquiti routers didn't support it so we had been using 2 all this time.  I will post back when I find out more.

[Richard Burts]

There are several points that I would like to address:

- there has been some discussion about whether this may be an issue with provider equipment. While we do not have much to rule this out I am dubious that this is due to provider equipment problems. 

- there has been some discussion about the age of the hardware and the software and whether these may be the problem. While it is true that they are old I am dubious that either of these is the cause of the problem.

- we have evidence that there are packets encrypted and packets decrypted. If there is two way traffic it seems that at least parts of the vpn are working ok.

- we have evidence that at least sometimes there are problems (encapsulation failed) with access to some local addresses.

I believe that we need some more investigation of the encapsulation failed messages and whether there are some times that this mac address does not appear in the arp table. Perhaps one way to test would be to coordinate local efforts with remote efforts to send traffic to the printers:

1) local ping to one or both printers.

2) show arp to verify that the mac address(es) are in the arp table.

3) attempt to send output to the printers

4) show arp to verify if the mac addresses are still in the arp table.

(and it might be good to have debug ip packet enabled during this to see if there are still encapsulation failed messages)

HTH

Rick

[wseyller]

I honestly don't think it is an issue with the IPS at all.  My client has ask myself and their ISP sale rep to look into everything before purchase of the new Fortigate.   I don't think anymore time will be spent on the existing equipment.  Too much time has been spent and it is not practical anymore.  Labor time is too great to not just by a better device and one that we will be included with support subscription from the manufacturer that I can leverage if needed.

The encapsulation error I have only seen that one time. 

This is why I personally think it may be the router that is the issue.  I have looked over the debugs alot lately and here is one that sticks out to me.  This one has been showing up continuously and I didn't really noticed this.   It shows that a packet from the source of the server was sent to this printer.  Then you see a packet from the printer being sent to the server IP address.  If you look it shows a gateway of the ISP gateway address.  To me that says it is getting routed to the internet instead of the IPSec tunnel.

*Oct 7 20:08:48.442: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), routed via RIB
*Oct 7 20:08:48.442: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), g=192.168.55.250, len 60, forward
*Oct 7 20:08:48.442: IP: tableid=0, s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), routed via RIB
*Oct 7 20:08:48.442: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), g=ISP-GATEWAY(HIDDEN), len 60, forward

[wseyller]

Also I want to add this in addition to my previous post showing the debug.

I created a lab of the same situation in GNS3 emulation with some C7200 Cisco Routers.   These routers are all running IOS 15.1 firmware.

You can see in the debug that there is IPSec written all over the debugs.  My previous post seems to make me think that it is not recognizing the interesting traffic properly and instead going out to the internet.  You will see in the debug at first an encapsulation error.  The first couple pings failed as ARP was taking place.  But this traffic passed thru just fine.  I copied and pasted these exact same cli config to another another lab of real physical cisco routers that included another old Cisco 1841 with IOS 12.4 and it gave me the same result symptoms of failure as the production environment.

Debugs from GNS3 lab:

*Oct  9 01:54:35.163: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct

R3#  9 01:54:35.167: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, NAT Outside(78), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:35.171: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:35.171: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), routed via RIB

*Oct  9 01:54:35.171: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, output feature, NAT Inside(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:35.171: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), g=192.168.55.250, len 84, forward

*Oct  9 01:54:35.175: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, encapsulation failed

*Oct  9 01:54:37.139: IP: s=2.2.2.2 (FastEthernet0/0), d=1.1.1.1, len 136, input f

R3#eature, packet consumed, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.151: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.155: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, NAT Outside(78), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.159: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.163: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), routed via RIB

*Oct  9 01:54:37.163: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, output feature, NAT Inside(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.163: IP: s=10.1.2.57

R3# (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), g=192.168.55.250, len 84, forward

*Oct  9 01:54:37.163: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, sending full packet

*Oct  9 01:54:37.171: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.175: IP: tableid=0, s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), routed via RIB

*Oct  9 01:54:37.175: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.175: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.175: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (Fast

R3#Ethernet0/0), len 84, output feature, packet consumed, IPSec: to crypto engine(75), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.183: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, pre-encap feature, IPSec Output Encap(1), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.183: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, pre-encap feature, Crypto Engine(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:37.187: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, sending full packet

*Oct  9 01:54:38.227: IP: s=2.2.2.2 (FastEthernet0/0), d=1.1.1.1, len 136, input feature, packet consumed, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.231: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, IPSec input

R3# classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.231: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, NAT Outside(78), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.231: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.231: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), routed via RIB

*Oct  9 01:54:38.235: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, output feature, NAT Inside(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.235: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), g=192.168.55.250, len 84, forward

*Oct  9 01:54:38.239: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, sending full packet

*Oct  9 01:5

R3#4:38.243: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.243: IP: tableid=0, s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), routed via RIB

*Oct  9 01:54:38.243: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.243: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.243: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, packet consumed, IPSec: to crypto engine(75), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.243: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (Fas

R3#tEthernet0/0), len 136, pre-encap feature, IPSec Output Encap(1), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.247: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, pre-encap feature, Crypto Engine(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:38.247: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, sending full packet

*Oct  9 01:54:39.291: IP: s=2.2.2.2 (FastEthernet0/0), d=1.1.1.1, len 136, input feature, packet consumed, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.295: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, IPSec input classification(49), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.299: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, NAT Outside(78), rtype 0, forus FALSE, sendself

R3# FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.299: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.299: IP: tableid=0, s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), routed via RIB

*Oct  9 01:54:39.299: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, output feature, NAT Inside(8), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.299: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), g=192.168.55.250, len 84, forward

*Oct  9 01:54:39.299: IP: s=10.1.2.57 (FastEthernet0/0), d=192.168.55.250 (FastEthernet0/1), len 84, sending full packet

*Oct  9 01:54:39.307: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57, len 84, input feature, MCI Check(92), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.311: IP: tableid=0, s=192.168.55.250 (FastEthern

R3#et0/1), d=10.1.2.57 (FastEthernet0/0), routed via RIB

*Oct  9 01:54:39.311: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.315: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, IPSec output classification(34), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.315: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), len 84, output feature, packet consumed, IPSec: to crypto engine(75), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.315: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, pre-encap feature, IPSec Output Encap(1), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.315: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len

R3# 136, pre-encap feature, Crypto Engine(3), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Oct  9 01:54:39.315: IP: s=1.1.1.1 (FastEthernet0/1), d=2.2.2.2 (FastEthernet0/0), len 136, sending full packet

R3#

[Richard Burts]

I believe that the debug that you post from the router is appropriate

*Oct 7 20:08:48.442: IP: s=192.168.55.250 (FastEthernet0/1), d=10.1.2.57 (FastEthernet0/0), g=ISP-GATEWAY(HIDDEN), len 60, forward

This represents the activity of the forwarding engine of the router. And as far as it is concerned forwarding to 10.1.2.57 should go out the outbound interface with the ISP as the next hop. As the packet gets to the outbound interface the crypto processing will recognize that this packet should be encrypted and sent out the vpn tunnel. That does not contradict the view of the forwarding engine. These are consistent: the forwarding engine sends the packet to the outbound interface and then the crypto processing takes over and processes the packet for the vpn. I believe that this debug shows that your router receives a packet from the server and forwards it to the printer and the printer then sends something to the server. It is not clear where or why the communication breaks down.

I am puzzled about why the config works in some lab situations and fails when you do it on real equipment. I believe that it suggests that there is something in the setup and operation of the real equipment that we have not yet recognized. 

If you and your management feel that you have put too much time into this and that you want to purchase new equipment, on which you expect to have better support, then this is a reasonable way to try to resolve this issue. If you do decide to try to solve the problem on existing equipment then post again to this discussion and we will continue to try to work with you to find a solution.

HTH

Rick

[wseyller]

I am going to try another lab in GNS3 with 12.4 firmware and see if the same issue occurs.  I am curious to know that.  Because that is one difference from the real equipment is that in GNS3 I am using routers with 15.1 IOS.