Cisco WRVS4400N v2 FW 2.0.21/Cisco 2951 IPSec Tunnels

interneturl · ‎06-22-2012

Hi,

We have an CISCO 2951 as a central hub in an IPSec VPN community, with six WRVS4400N branch office routers connecting into it.

Setting up the VPN tunnels worked fine, except after a while the tunnels seem to disconnect all by themselves, and they will not reconnect. Browsing the Cisco WRVS4400N logs we get:

[VPN Log]: ERROR: "Taller-182": pfkey write() of SADB_X_DELFLOW message 16 for flow int.0@0.0.0.0 failed. Errno 14: Bad address

If I restart the WRVS4400N , the VPN connects just fine. If I let it sit for a while (like an hour or so) and hit connect, it connects just fine as well. Furthermore, if I enter the configuration screen for the VPN tunnel on the WRVS4400N , and hit SAVE (make no changes) it also connects. Just over time it seems to disconnect, and will not reconnect without a restart.

Can anyone enlighten me to a source of the problem??

Jun 22 10:08:01 - [VPN Log]: "Taller-182" #1: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
Jun 22 10:08:01 - [VPN Log]: "Taller-182" #1: STATE_MAIN_I3: sent MI3, expecting MR3
Jun 22 10:08:02 - [VPN Log]: | protocol/port in Phase 1 ID Payload is 17/0. accepted with port_floating NAT-T
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #1: Main mode peer ID is ID_IPV4_ADDR: '190.3.108.131'
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #1: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #1: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_md5 group=modp1024}
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #2: initiating Quick Mode PSK+ENCRYPT+TUNNEL+DISABLEARRIVALCHECK+UP {using isakmp#1}
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #3: initiating Quick Mode PSK+ENCRYPT+TUNNEL+DISABLEARRIVALCHECK+UP {using isakmp#1}
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #2: ignoring informational payload, type IPSEC_RESPONDER_LIFETIME
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #2: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #2: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0x8af3b4bd <0x9ebd59af xfrm=3DES_0-HMAC_SHA1 NATD=190.3.108.131:4500 DPD=none}
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #3: ignoring informational payload, type IPSEC_RESPONDER_LIFETIME
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #3: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
Jun 22 10:08:02 - [VPN Log]: "Taller-182" #3: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0x1d42f1c9 <0x9ebd59b0 xfrm=3DES_0-HMAC_SHA1 NATD=190.3.108.131:4500 DPD=none}
Jun 22 10:09:14 - [VPN Log]: "Taller-182" #4: initiating Main Mode
Jun 22 10:09:14 - [VPN Log]: "Taller-182" #4: received Vendor ID payload [RFC 3947] method set to=109
Jun 22 10:09:14 - [VPN Log]: "Taller-182" #4: enabling possible NAT-traversal with method 3
Jun 22 10:09:14 - [VPN Log]: "Taller-182" #4: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
Jun 22 10:09:14 - [VPN Log]: "Taller-182" #4: STATE_MAIN_I2: sent MI2, expecting MR2
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: received Vendor ID payload [Cisco-Unity]
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: received Vendor ID payload [Dead Peer Detection]
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: ignoring unknown Vendor ID payload [25bc71307e46d7adbdc6cedd8a3dea1e]
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: received Vendor ID payload [XAUTH]
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: I did not send a certificate because I do not have one.
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: NAT-Traversal: Result using 3: i am NATed
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: STATE_MAIN_I3: sent MI3, expecting MR3
Jun 22 10:09:15 - [VPN Log]: | protocol/port in Phase 1 ID Payload is 17/0. accepted with port_floating NAT-T
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: Main mode peer ID is ID_IPV4_ADDR: '190.3.108.131'
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #4: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_md5 group=modp1024}
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #5: initiating Quick Mode PSK+ENCRYPT+TUNNEL+DISABLEARRIVALCHECK+UP {using isakmp#4}
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #5: ignoring informational payload, type IPSEC_RESPONDER_LIFETIME
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #5: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
Jun 22 10:09:15 - [VPN Log]: "Taller-182" #5: STATE_QUICK_I2: sent QI2, IPsec SA established {ESP=>0x740585e6 <0x9ebd59b1 xfrm=3DES_0-HMAC_SHA1 NATD=190.3.108.131:4500 DPD=none}
Jun 22 10:13:58 - [VPN Log]: shutting down
Jun 22 10:13:58 - [VPN Log]: forgetting secrets
Jun 22 10:13:58 - [VPN Log]: "Taller-182": deleting connection
Jun 22 10:13:58 - [VPN Log]: "Taller-182" #5: deleting state (STATE_QUICK_I2)
Jun 22 10:13:58 - [VPN Log]: ERROR: "Taller-182" #5: pfkey write() of SADB_X_ADDFLOW message 29 for flow %trap failed. Errno 14: Bad address
Jun 22 10:13:58 - [VPN Log]: | 02 0e 00 0b 17 00 00 00 1d 00 00 00 03 0a 00 00
Jun 22 10:13:58 - [VPN Log]: | 03 00 01 00 00 00 01 04 00 00 00 00 02 00 00 00
Jun 22 10:13:58 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 02 00 00 00 c0 a8 01 22 00 00 00 00 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 00 00 00 00 00 00 00 00 03 00 15 00 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 02 00 00 00 c0 a8 b6 00 00 00 00 00 84 0b 00 40
Jun 22 10:13:58 - [VPN Log]: | 03 00 16 00 00 00 00 00 02 00 00 00 c0 a8 fe 00
Jun 22 10:13:58 - [VPN Log]: | b0 25 01 00 22 00 00 00 03 00 17 00 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 02 00 00 00 ff ff ff 00 3a 20 64 65 6c 65 74 69
Jun 22 10:13:58 - [VPN Log]: | 03 00 18 00 00 00 00 00 02 00 00 00 ff ff ff 00
Jun 22 10:13:58 - [VPN Log]: | 54 45 5f 51 00 00 00 00
Jun 22 10:13:58 - [VPN Log]: | 02 04 00 03 0b 00 00 00 1e 00 00 00 03 0a 00 00
Jun 22 10:13:58 - [VPN Log]: | 03 00 01 00 74 05 85 e6 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 c0 a8 01 22 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 be 03 6c 83
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 04 00 03 0b 00 00 00 1f 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 01 00 9e bd 59 b1 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 be 03 6c 83 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 c0 a8 01 22
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: "Taller-182" #3: deleting state (STATE_QUICK_I2)
Jun 22 10:13:59 - [VPN Log]: | 02 04 00 03 0b 00 00 00 20 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 01 00 1d 42 f1 c9 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 c0 a8 01 22 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 be 03 6c 83
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 04 00 03 0b 00 00 00 21 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 01 00 9e bd 59 b0 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 be 03 6c 83 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 c0 a8 01 22
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: "Taller-182" #2: deleting state (STATE_QUICK_I2)
Jun 22 10:13:59 - [VPN Log]: | 02 04 00 03 0b 00 00 00 22 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 01 00 8a f3 b4 bd 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 c0 a8 01 22 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 be 03 6c 83
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 04 00 03 0b 00 00 00 23 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 01 00 9e bd 59 af 00 01 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | ff ff ff ff 00 00 00 00 03 00 05 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 be 03 6c 83 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 06 00 00 00 00 00 02 00 00 00 c0 a8 01 22
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: "Taller-182" #4: deleting state (STATE_MAIN_I4)
Jun 22 10:13:59 - [VPN Log]: "Taller-182" #1: deleting state (STATE_MAIN_I4)
Jun 22 10:13:59 - [VPN Log]: ERROR: "Taller-182": pfkey write() of SADB_X_DELFLOW message 36 for flow int.0@0.0.0.0 failed. Errno 14: Bad address
Jun 22 10:13:59 - [VPN Log]: | 02 0f 00 0b 0e 00 00 00 24 00 00 00 03 0a 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 15 00 00 00 00 00 02 00 00 00 c0 a8 b6 00
Jun 22 10:13:59 - [VPN Log]: | 00 00 00 00 84 0b 00 40 03 00 16 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 c0 a8 fe 00 b0 25 01 00 22 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 03 00 17 00 00 00 00 00 02 00 00 00 ff ff ff 00
Jun 22 10:13:59 - [VPN Log]: | a8 eb ff bf 00 00 00 00 03 00 18 00 00 00 00 00
Jun 22 10:13:59 - [VPN Log]: | 02 00 00 00 ff ff ff 00 00 00 00 00 00 00 00 00

morov12345 · ‎06-25-2012

I have exact the same problem as you.

I even did contact Cisco but the first level technician was unable to sole it and his answer was that the problem is at the other side where I have Linux firewall (pfSense).

Probably you can open a case with Cisco since you have two Cisco devices... The support won't be able to reject service.

Tom Watts · ‎07-02-2012

Hi guys, is there any additional symptoms reported aside the tunnel disconnecting such as the router locking up, WAN connectivity lost, slow throughput, etc? When the tunnel drops, are you able to set a packet capture from the 2951 to see what error is coming back when the negotiation requests are being sent? Is the 2951 using any kind of dynamic policy or any kind of NAT-T? Are there any DPD or purge messages?

-Tom Please mark answered for helpful posts http://blogs.cisco.com/smallbusiness/

morov12345 · ‎07-03-2012

Hello Thomas,

In my case, it was pfSense Linux firewall connected to the WRVS4400N...

We changed the ISP modem during the troubleshooting and as a result, the tunnel was always up at both sides. In both modems (the original modem and the new one) we used PPPoE (bridge mode) so there was no reason to have a problem with the first one...

Unfortunately, with the new modem we got another problem:

- If I try to connect from the pfSense network to the WRVS4400N network I have access all the time

- If I try to connect from the WRVS4400N network to the pfSense network I am getting "Request timed out". If I do PING x.x.x.x -t for a minute I am getting a reply and the connection works fine. As soon as I stop using the tunnel for more then 5 minutes, the WRVS4400N shows that the tunnel is up but the ping shows again "Request timed out".

As a final solution, we replaced the WRVS4400N with RV-042 and now it works fine at both sides all the time...

So... Sorry but I am done with the WRVS4400N. Do not have time for it.