Re: Load balancing stopped working after upgrading to 3.6.7F

astrand · ‎05-08-2003

Hi,

I just upgraded our two VPN 3000 concentrators to 3.6.7F to fix the security vulnerabilities. However, after upgrading I am no longer able to access through the virtual IP address. Access through both real IP address works, but attempts to the virtual address does not even appear in the logs. In the load balancing statistics I can see that both concentrators are aware of each other and both agree who is the master and who is secondary.

The client I use is 3.6.3(B) running on Windows 2000. We use IPSec over TCP port 10000 to access the concentrators.

Any help would be appreciated!

Regards,

Harald

jfrahim · ‎05-08-2003

I am not sure if you are using Certificates or preshared keys, but there was a bug in the cocnentrator if you were using LB and certs

I believe the BUG ID was: CSCdz85885

If you are not hitting this bug, then you would need to collect the debugs from all the concentrators involved in loadbalancing and the client

Hope that helps

Jazib

w.tresp · ‎05-08-2003

Hi,

I think Bug-ID CSCdz85885 describes a problem which occurs if load balance notify packet arrives at the client before the certificate packet with certificate arrives.

Thats not the problem. We run into the same problem with the VPN Concentrator Release 4.0.1 using preshared keys and IPSec over TCP. It seems that it is not possible to connect to the virtual ip address of the load balanced cluster members.

The VPN client 4.0 times out and shows an error message like "cannot establish TCP connection".

The VPN3000 master shows nothing in its logs. If you try to connect directly to the real IP address of the cluster members everything works fine.

We had to downgrade to the 4.0 release.

Regards,

Wilfried Tresp

astrand · ‎05-09-2003

Hi again,

I also use preshared keys, so I assume this is a new issue for version 3.6.7F and 4.0.1. It seemed to me like the concentrators were no longer listening for the virtual address on the external interface.

In order to temporarly solve the problem and allow users to connect I disabled load balancing and changed the real address for one of the concentrators to the virtual address.

When I returned this morning, I configured a new address for load balancing and enabled it on the concentrators (while our users still connect to the real address). To my surprise my test clients are now able to connect to the new virtual address.

I am not sure if I dare to change the configuration back to use LB for the real users. It would be preferable to first have an explanation why load balancing did not work in the first place. Since we were upgrading from 3.6.7, I did not expect any problems.

Regards,

Harald

engel · ‎05-09-2003

Just tried the load-balance feature with Concentrator VPN3060 loaded with version 3.6.7F.

I didn`t have any problem connecting to the virtual IP address. The client can be redirected to the Secondary by the Master of the cluster. The connection can be completed without any problem. But since these devices are in a lab`s network, the real production devices may behave differently.

PS: The client is version 3.6.4 on a Windows 2000 Proffesional Japanese.

w.tresp · ‎05-09-2003

Hi Harald,

maybe it helps to define a filter which allows the destination port tcp/10000 to the virtual ip address of your vpn3000?

It seems that the cluster ip address was forgotten by the developers of the security bug fix.

Regards,

Wilfried Tresp

engel · ‎05-09-2003

Hi,

Just tried with IPSec over TCP, IPSec over UDP . Both can be load-balanced by the Master of the Cluster. I don`t need to define a rule on the public filter to allow TCP port 10000 or UDP port 10000. The default filter just work fine.

Here attached is the log of the client when connecting to the virtual IP address (10.10.10.3) . The Master is 10.10.10.2 and the Secondary is 10.10.10.1

1 23:00:40.623 05/09/03 Sev=Info/6 DIALER/0x63300002

Initiating connection.

2 23:00:40.773 05/09/03 Sev=Info/6 IKE/0x6300003B

Attempting to establish a connection with 10.10.10.3.

3 23:00:40.803 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK AG (SA, KE, NON, ID, VID, VID, VID) to 10.10.10.3

4 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.3

5 23:00:40.953 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK AG (SA, KE, NON, ID, HASH, VID, VID, VID, VID, VID) from 10.10.10.3

6 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 12F5F28C457168A9702D9FE274CC0100

7 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000001

Peer is a Cisco-Unity compliant peer

8 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 09002689DFD6B712

9 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000001

Peer supports XAUTH

10 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = AFCAD71368A1F1C96B8696FC77570100

11 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000001

Peer supports DPD

12 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 4048B7D56EBCE88525E7DE7F00D6C2D3C0000000

13 23:00:40.953 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 1F07F70EAA6514D3B0FA96542A500306

14 23:00:40.983 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK AG *(HASH, NOTIFY:STATUS_INITIAL_CONTACT) to 10.10.10.3

15 23:00:40.983 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.3

16 23:00:40.983 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK INFO *(HASH, NOTIFY:LOAD_BALANCE) from 10.10.10.3

17 23:00:40.983 05/09/03 Sev=Info/5 IKE/0x63000017

Marking IKE SA for deletion (COOKIES = F4142F0A4B48020D 2E314A47DEDCDE2D) reason = DEL_REASON_LOAD_BALANCING

18 23:00:40.983 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK INFO *(HASH, DEL) to 10.10.10.3

19 23:00:41.124 05/09/03 Sev=Info/6 IKE/0x6300003B

Attempting to establish a connection with 10.10.10.1.

20 23:00:41.154 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK AG (SA, KE, NON, ID, VID, VID, VID) to 10.10.10.1

21 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x6370001F

TCP SYN sent to 10.10.10.3, src port 3720, dst port 10000

22 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x6370001C

TCP SYN-ACK received from 10.10.10.3, src port 10000, dst port 3720

23 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x63700020

TCP ACK sent to 10.10.10.3, src port 3720, dst port 10000

24 23:00:41.274 05/09/03 Sev=Info/4 IPSEC/0x63700014

Deleted all keys

25 23:00:41.274 05/09/03 Sev=Info/4 IPSEC/0x63700012

Delete all keys associated with peer 10.10.10.3

26 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x63700022

TCP RST sent to 10.10.10.3, src port 3720, dst port 10000

27 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x6370001F

TCP SYN sent to 10.10.10.1, src port 3720, dst port 10000

28 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x6370001C

TCP SYN-ACK received from 10.10.10.1, src port 10000, dst port 3720

29 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x63700020

TCP ACK sent to 10.10.10.1, src port 3720, dst port 10000

30 23:00:41.274 05/09/03 Sev=Info/4 IPSEC/0x63700014

Deleted all keys

31 23:00:41.274 05/09/03 Sev=Info/6 IPSEC/0x6370002B

Sent 3 packets, 0 were fragmented.

32 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.1

33 23:00:41.314 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK AG (SA, KE, NON, ID, HASH, VID, VID, VID, VID, VID) from 10.10.10.1

34 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 12F5F28C457168A9702D9FE274CC0100

35 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000001

Peer is a Cisco-Unity compliant peer

36 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 09002689DFD6B712

37 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000001

Peer supports XAUTH

38 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = AFCAD71368A1F1C96B8696FC77570100

39 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000001

Peer supports DPD

40 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 4048B7D56EBCE88525E7DE7F00D6C2D3C0000000

41 23:00:41.314 05/09/03 Sev=Info/5 IKE/0x63000059

Vendor ID payload = 1F07F70EAA6514D3B0FA96542A500306

42 23:00:41.344 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK AG *(HASH, NOTIFY:STATUS_INITIAL_CONTACT) to 10.10.10.1

43 23:00:41.354 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.1

44 23:00:41.354 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 10.10.10.1

45 23:00:43.447 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 10.10.10.1

46 23:00:43.747 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.1

47 23:00:43.747 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 10.10.10.1

48 23:00:43.757 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 10.10.10.1

49 23:00:44.078 05/09/03 Sev=Info/5 IKE/0x6300005D

Client sending a firewall request to concentrator

50 23:00:44.078 05/09/03 Sev=Info/5 IKE/0x6300005C

Firewall Policy: Product=Cisco Integrated Client, Capability= (Centralized Protection Policy).

51 23:00:44.088 05/09/03 Sev=Info/4 IKE/0x63000013

SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 10.10.10.1

52 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x6300002F

Received ISAKMP packet: peer = 10.10.10.1

53 23:00:44.919 05/09/03 Sev=Info/4 IKE/0x63000014

RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 10.10.10.1

54 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x63000010

MODE_CFG_REPLY: Attribute = INTERNAL_IPV4_ADDRESS: , value = 192.168.1.200

55 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x63000010

MODE_CFG_REPLY: Attribute = INTERNAL_IPV4_NETMASK , value = 255.255.255.0

56 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x6300000D

MODE_CFG_REPLY: Attribute = MODECFG_UNITY_SAVEPWD: , value = 0x00000000

57 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x6300000D

MODE_CFG_REPLY: Attribute = MODECFG_UNITY_PFS: , value = 0x00000000

58 23:00:44.919 05/09/03 Sev=Info/5 IKE/0x6300000E

MODE_CFG_REPLY: Attribute = APPLICATION_VERSION, value = Cisco Systems, Inc./VPN 3000 Concentrator Version 3.6.7.F built by vmurphy on Apr 25 2003 11:02:30

--------------truncated------------------

astrand · ‎05-09-2003

I decided to take the risk and go ahead and put the load balancing back for the users and it seems to be working fine. Still wondering why it was not working before though...

Thank you all for your help!

Regards,

Harald

astrand · ‎05-12-2003

Hi again,

This weekend we had a power failure in the data centre and both concentrators were thus rebooted. When they came up again, the load balancing did not start.

Also, we discovered that IPSec over UDP does not work for us at all when using 3.6.7.F. Not even if we use the real addresses. The user is able to authenticate, but cannot access any resources. The in- and out counters for the session remains zero.

In order to solve both these issues I downgraded the concentrators to 3.6.7 and now everything is working again.

Any information about these issues in 3.6.7.F and a future release with fixes would be appreciated.

Regards,

Harald

engel · ‎05-12-2003

Hi Harald,

I just discovered that a "VRRP" setting of VPN3000 will not work with VPN Client that using "IPSec over TCP".

Would you clarify again what do you mean with load balancing of VPN3000 ?

If it is a load-balance setting (both of VPN3000 is Active-Active) than an IPSec over TCP client will not have problem connecting to the Concentrator.

If it is a VRRP setting (one of VPN3000 is Active, the other is Standby) than an IPSec over TCP client can not make a connection to the Master device that was a standby one.

The workaround is to create a rule on the public interface of the standby device to permit TCP/10000.

Anyway I don`t like this solution, and would like Cisco to verify this thing.

Anyone else can confirm this defect of version 3.6.7F ??

Best Regards,

Engel

astrand · ‎05-12-2003

Hi Engel,

We are using the Active-Active load balancing, i.e. we are not using VRRP.

Regards,

Harald

jfrahim · ‎05-12-2003

Make Sure that you are not running into:

CSCea09329

Load balancing doesnt work for ctcp connections

Jazib

astrand · ‎05-12-2003

I cannot find that bug number in the bug toolkit. Could you provide more information about the bug?

Also, do you have any information if there is also a known bug with IPSec over UDP for 3.6.7.F?

Thanks in advance!

Harald

engel · ‎05-18-2003

Hi Harald,

Are you able to contact Cisco TAC about 3.6.7.F defect ?

Just want to know if there is any progress with the issue of 3.6.7.F`s load-balance problem.

astrand · ‎05-19-2003

I have not heard anything about any fixes yet. Hopefully there will be a 3.6.7.G soon that will fix both the IPSec over TCP and the IPSec over UDP issue.