Cisco ISR with site to site VPN. Tunnel is up but traffic will not pass.

[wseyller]

Using Cisco ISR 1841

I can see some traffic from the IPSec VPN on the wan interface when the other side tries to ping to printers on the local lan.  There is no traffic from the VPN on the lan side.  The tunnel shows to be up on both sides.  The other side is using a fortigate firewall in a datacenter.

Here is my configuration.  The server at 10.1.2.57/32 is unable to ping a printer or anything else for example at 192.168.55.250.

I am not listing any acl for the wan interface because I removed it anyway during this testing.  Also I am list fake public ip addresses to censor.

version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable secret 5 ************************
!
no aaa new-model
!
resource policy
!
clock timezone EST -5
clock summer-time EST recurring 2 Sun Mar 2:00 1 Sun Nov 2:00
ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.55.1 192.168.55.99
ip dhcp excluded-address 192.168.55.150 192.168.55.254
!
ip dhcp pool TASK55
network 192.168.55.0 255.255.255.0
default-router 192.168.55.1
domain-name somedomain.int
dns-server 192.168.1.10 192.168.55.1
!
!
no ip domain lookup
ip domain name somedomain.int
ip name-server 192.168.1.10
!
!
crypto pki trustpoint TP-self-signed-25944030
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-25944030
revocation-check none
rsakeypair TP-self-signed-25944030
!
!
crypto pki certificate chain TP-self-signed-25944030
certificate self-signed 01
**************************************
quit
username ais privilege 15 password 7 ******************
!
!
crypto isakmp policy 2
encr 3des
authentication pre-share
group 2
crypto isakmp key SecretPass address 2.2.2.2
crypto isakmp keepalive 3600
crypto isakmp aggressive-mode disable
!
crypto ipsec security-association lifetime seconds 28800
!
crypto ipsec transform-set TASK_TS esp-3des esp-sha-hmac
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
description Tunnel to2.2.2.2
set peer 2.2.2.2
set transform-set TASK_TS
set pfs group2
match address 100
!
!
!
!
interface Tunnel0  (NOT RELATED TO IPSEC TUNNEL)
ip address 10.0.0.6 255.255.255.252
ip mtu 1476
tunnel source 1.1.1.1
tunnel destination 10.10.10.10
!
interface FastEthernet0/0
ip address 1.1.1.1 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
no mop enabled
crypto map SDM_CMAP_1
!
interface FastEthernet0/1
ip address 192.168.55.1 255.255.255.0
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
router ospf 1
router-id 192.168.55.1
log-adjacency-changes
network 192.168.55.0 0.0.0.255 area 0

network 10.0.0.4 0.0.0.3 area 0
!
!
!
ip http server
ip http authentication local
ip http secure-server
ip nat inside source list NATLIST interface FastEthernet0/0 overload
!
ip access-list extended NATLIST
deny ip 192.168.55.0 0.0.0.255 host 10.1.2.57
permit ip 192.168.55.0 0.0.0.255 any
!
access-list 100 remark CCP_ACL Category=4
access-list 100 remark IPSec Rule
access-list 100 permit ip 192.168.55.0 0.0.0.255 host 10.1.2.57
!
!
!
!
!
!
control-plane
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
exec-timeout 0 0
privilege level 15
logging synchronous
login local
transport input ssh
line vty 5 15
exec-timeout 0 0
logging synchronous
login local
transport input ssh
!
scheduler allocate 20000 1000
ntp clock-period 17179132
end

[Richard Burts]

I will be interested in the results of your lab testing for a difference in version. And I will be very surprised if different versions have different results. Please let us know when you have results.

I have been wondering about the issue with the live equipment and wonder if you would have them attempt to print something and immediately show the content of the address translation table and post that output.

HTH

Rick

[wseyller]

So I did lab in GNS3 an IPsec connection with two routers running 12.4.  But these were Cisco 3725 routers as that is what I had because my 1841 firmware wasn't supported it seems.  It worked fine.  Although the NAT configuration wouldn't take without bumping the ram up from 128mb to something higher like 256 MB.  So not sure that 12.4 is a problem but maybe it is something else with these particular 1841's.  Maybe they need more RAM, but whatever, who knows.

So I received the new Fortigate Firewall.  I have dealt with them a little before but I am still somewhat new to them.  Got a year of Forticare support in case I have trouble.  I upgraded the firmware and have got it all setup I think.  I tried the same VPN configuration to a physical Cisco 2821 router that I have running 15.1 IOS.  The VPN works perfectly between the two and they can pass LAN traffic back and forth.  Almost seemed to easy.  One of the IPSec options in the fortigate is to choose that your connecting to a cisco device or another fortigate.  I left it on the fortigate setting but it still worked with the cisco device.  Not sure what that options really does in the background.

I am taking this to the customer location tomorrow morning.  Hope all goes well.  I will report back afterwards.

[Richard Burts]

I hope it does work better with the Fortigate. Please let us know the outcome.

HTH

Rick

[wseyller]

Still doesn't work.  Exact same symptoms.  I even had them remote in and take a look at the firewall on my side and they thought the configuration looked correct.  They told me they would talk to their supervisor and call me back later.

[Richard Burts]

Sorry to hear that the new hardware did not solve the problem. Interesting that it is still the same symptoms. That sort of fits with our observation that your router config seemed to be ok. And with the fact that there was two way traffic on the vpn. 

Am I correct in understanding that this site to site vpn is to carry traffic to a printer at your site? I am wondering if there might be some issue with the setup of the printer that causes it to be incompatible with the server? Could you have the server send some type of traffic to some other device in your network and see if that works better?

HTH

Rick

[wseyller]

Yeah Ive  been trying pings to different computers this whole time too.  At this point the only thing I can possibly tell the client to have another internet circuit added.  The router I am peering with I have no control of that.  

I tested this device at home with the same config to a Cisco router I had.  I used another router in the middle to similate the ISP gateways.  It worked easily without any fuss.

[Richard Burts]

Thanks for confirming that you have been testing with ping to other devices on your subnet from the server. Am I correct in understanding that these ping tests failed? It makes me wonder about the possibility that the problem may be on the other end.

HTH

Rick

[wseyller]

Yes pings from either end fail. What is strange is there is some traffic
going to the printers on port 9100 which is a print job. Like there is a
job stuck in the print queue on the server because it keeps trying. The
printer receives the tcp traffic. But printer tries to send an ack back
and it can't make it back to the other side. Can see this in the packet
captures. It repeats over and over and this has been seen on the others
routers I tried. But if you ping the printer nothing at all. Almost like
they allow only destination port 9100 outbound but nothing else in, which
sounds crazy to me.

I don't assume it's setup right on their end. I can't. But they think it
is.

My Cisco router with 15.1 IOS I may try this weekend to connect to the
fortigate with a VPN. I connected them successfully with both devices in
my home. This time through a real internet connection. Then I will also
peer with one of the other three sites that are working just fine. If I
can make it work with fortigate then I think there is no ISP issue. If it
fails but I can connect to the other sites then maybe issue with ISP.

[wseyller]

So I ran some other test.  At home I have a Cisco 5510 ASA.  I setup ipsec connections with similiar settings to the problem site and also the customers other 3 sites.

The problem site I get the same thing.  I can bring the tunnel up but no traffic with pass.  With the other sites I can pass traffic to and from either end.

So ISP issue?  Only thing I can think of.

[Georg Pauwen]

Hello,

can you post the full configs of the routers you have configured in GNS3 ?

[Richard Burts]

It is interesting that in your test using your ASA that it is successful with other sites and has the same kind of issue with the problem site. In this testing you have the ASA configured like the firewall at the remote side? And do you have a device connected to the ASA which uses the IP address of the remote server?

HTH

Rick

[wseyller]

I used the same settings I could. LAN addressing on each side, encryption
DH group and all. Only difference is my public IP at my house that I
peered to is different.

[Richard Burts]

This is quite puzzling. Am I correct in remembering from various posts that you had this problem using a router at the site for the vpn and then replaced that router with a Fortigate and still had exactly the same symptom? If two different platforms at the site had the same issue it would sort of point toward the issue being at the other end.

But now you have tested using an ASA for the other end and still have the same issue. Seems unlikely that if the problem were on the original remote peer that you would create the same issue on your ASA.

Is my memory of the various steps and their outcome accurate?

When you were testing with your ASA did you do any show crypto IPSec sa on either or both sides? If so could you post those outputs?

Would you post the output of show access-list from your router?

HTH

Rick

[wseyller · Accepted solution]

So the ISP made a change that now allows the VPN to work.  Not sure exactly what they did but heard it was something in regards to reflexive ACLs.

[Richard Burts]

Thank you for posting back and telling us that the ISP made some change and that change solved the problem. It is certainly nice to see this question finally marked as solved.

HTH

Rick