redundancy issue with CSS11150

[kope]

I have two CSS11150 set up as box-to-box redundant config. the software version is 6.10 build 405. It has been running fine for for quit some time (6 months).

Yesterday, the primary suddenly failed over to the backup for no reason. the primay was still running. but when i did a "sh redundancy", both were "master"; and it started loosing ping packets randomly. I looked at the log and it shown "duplicate ip addresses".

Do this product has a inherit problem like this, I am using the 610.405 software. Is there a bug on this type of hardware/software combination that has this kind of problem?

thanks.

[pknoops]

Hi,

In general, there are really only 2 reasons this senario can occur.

1. The physical link between the 2 boxes went down. This would cause the backup to take over as master as it would stop receiving heartbeats from the MASTER and you would have dual mastership.

2. The MASTER could be getting slammed with high traffic and therefore would not be able to process heartbeats from the BACKUP and therefore the backup would think the MASTER is down and then take over leaving the other MASTER in tact and again have a dual mastership situation.

Check the sys.log and see if the phy link went down or see if there is anything else in there that could help.

Pete..

[kope]

AUG 31 18:15:04 5/1 101 VRRP-4: Virtual router 128 on interface 20.1.1.2 entering into VRRP negotiation

AUG 31 18:15:07 5/1 102 VRRP-4: Virtual router 128 on interface 20.1.1.2 exiting out of VRRP negotiation

AUG 31 18:15:14 5/1 103 VRRP-4: Virtual router 128 on interface 20.1.1.2 entering into VRRP negotiation

AUG 31 18:15:15 5/1 104 NETMAN-2: Enterprise:Service Transition:Leo1 -> down

AUG 31 18:15:15 5/1 105 NETMAN-2: Enterprise:Service Transition:ursa1 -> down

AUG 31 18:15:15 5/1 106 NETMAN-2: Enterprise:Service Transition:taurus1 -> down

AUG 31 18:15:16 5/1 107 VRRP-4: Virtual router 128 on interface 20.1.1.2 exiting out of VRRP negotiation

AUG 31 18:15:35 5/1 108 NETMAN-2: Enterprise:Service Transition:Leo1 -> down

AUG 31 18:15:41 5/1 109 APP-4: APP: Peer IP address 20.1.1.1 session going to state APP_SESSION_INIT

AUG 31 18:15:55 5/1 110 VRRP-4: Virtual router 128 on interface 20.1.1.2 entering into VRRP negotiation

AUG 31 18:15:57 5/1 111 VRRP-4: Virtual router 128 on interface 20.1.1.2 exiting out of VRRP negotiation

AUG 31 18:15:59 5/1 112 VRRP-4: Virtual router 128 on interface 20.1.1.2 entering into VRRP negotiation

AUG 31 18:16:01 5/1 113 VRRP-4: Virtual router 128 on interface 20.1.1.2 exiting out of VRRP negotiation

AUG 31 18:16:27 5/1 114 IPV4-4: Duplicate IP address detected: 1xx.38.54.65 00-08-e2-10-f4-40

AUG 31 18:16:27 5/1 115 IPV4-4: Incoming CE 0x2001f00, incoming (0 based) SLP 0x8

AUG 31 18:16:27 5/1 116 IPV4-4: Duplicate IP address detected: 1xx.38.54.65 00-08-e2-10-f4-40

AUG 31 18:16:27 5/1 117 IPV4-4: Incoming CE 0x2001f00, incoming (0 based) SLP 0x8

AUG 31 18:16:34 5/1 118 IPV4-4: Duplicate IP address detected: 1xx.38.54.65 00-08-e2-10-f4-40

I cut a log.sys here. Any thought?

[pknoops]

Not based on that. Anything before that. Looks like the APP session was having an issue as well which would indicate lack of comm between the boxes.

[kope]

below are the events before that. thanks.

the problem began aug 31 18:15:04

AUG 26 11:54:49 5/1 70 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 26 11:54:54 5/1 71 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 26 11:54:57 5/1 72 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 26 11:55:00 5/1 73 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 28 06:13:20 5/1 74 NETMAN-2: Enterprise:Service Transition:taurus1 -> down

AUG 28 06:13:20 5/1 75 NETMAN-2: Enterprise:Service Transition:ursa1 -> down

AUG 28 06:13:21 5/1 76 NETMAN-2: Enterprise:Service Transition:Leo1 -> down

AUG 30 09:51:33 5/1 77 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 09:51:35 5/1 78 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 09:51:39 5/1 79 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 09:51:44 5/1 80 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 09:51:47 5/1 81 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 09:51:51 5/1 82 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:37 5/1 83 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:40 5/1 84 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:44 5/1 85 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:49 5/1 86 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:52 5/1 87 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 30 10:09:56 5/1 88 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:09 5/1 89 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:11 5/1 90 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:15 5/1 91 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:51 5/1 92 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:54 5/1 93 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 10:01:57 5/1 94 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:22 5/1 95 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:24 5/1 96 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:28 5/1 97 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:34 5/1 98 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:36 5/1 99 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 14:01:40 5/1 100 NETMAN-1: TRAP:Authentication:Generated by: 168.38.70.34

AUG 31 18:15:04 5/1 101 VRRP-4: Virtual router 128 on interface 20.1.1.2 entering into VRRP negotiation

[pknoops · Accepted solution]

At this point based on the data seen here, it would appear that this was more than likely related to one CSS being hit hard with traffic and not being able to respond to heartbeats forcing a new VRRP negotiation. The other thing i would recommend if you feel you need to pursue this even further would be to turn up the logging on the CSS.

Pete..