commit_VipRedundConfig problems.

nmfoxton · ‎05-22-2012

Sorry if this has been discussed before, but the search turned up nothing.

We have several pairs of CSS11501 and 11503 in our network.

This issue affects only one pair of CSS11503 in one of our data centres.

We run the same version of software throughout the estate;

CNMLW-CSS-1A# sh ver

Version: sg0820402 (08.20.4.02)

Flash (Locked): 08.10.1.06

Flash (Operational): 08.20.4.02

Type: PRIMARY

Licensed Cmd Set(s): Standard Feature Set

We use vrrp in one-armed mode for load balancing and they units have performed great for a number of years. We're obviously going to be migrating to ACE ... but not just yet.

We have started to experience a problem with replicating the configurations between two CSS11503 in a pair.

When running the commit-VipRedundConfig, it starts off happily enough, though slowly.

Ending with "working" and the spinning cursor, even after 1 hour the script hasn't completed.

We noted on the backup CSS that the APP configuration disappears during the process and I can't remember if this is normal behaviour.

Re-adding the app session configuration seems to interrupt the process, and when checking the configuration on the backup CSS approximately half of it is missing. Everything after the first owner is gone. Obviously not a happy situation.

I have left the script running overnight tonight to see if it finally completes ... we'll see in the morning how it gets on.

Theories ;

1. Configuration is too large, or just large enough to make the commit script take too long for realistic service.

2. Software bug?

3. Combination of both.

4. From now on manually add config to both CSS's and maintain it by process management.

Any clues what the problem is?

Jorge Bejarano · ‎05-22-2012

Hi,

Did you run it with -d? ( It can show more details)

Did you notice any error?

What is the uptime of the devices? Could you try to reboot the backup?

Jorge

nmfoxton · ‎05-23-2012

Thankyou Jorge,

Uptime is 193 days.

Overnight the script has not completed and half the configuration is still missing on the backup unit.

I have had to "stop" the script using "-f"

I've now initiated the commit again using "-d"

I'll leave the session open and come back to it later, I'd rather not reboot the backup unit, but if this fails again I have little choice but to try it. Response so far is;

CNMLW-CSS-1A# commit_VipRedundConfig "-d local 172.17.199.4 remote 172.17.199.5"

CNMLW-CSS-1A#

Verifying the IP Format for the passed in ip addresses

Checking available disk space on systems ...

Checking the disk space locally before continuing with the script.

Verifying that another local session is not running the script.

Creating script/vipr_config_sync_lock file.

Verifying app and redundancy configs ...

Verifying that app session is up with backup switch.

Making sure app session is up.

Seconds to wait before calling it quits: 60

Checking the disk space remotely before continuing with the script.

Checking local and remote switch versions ...

Storing the running code versions of the local and remote switch.

Storing the local switch's version.

Retrieving the remote switch's version.

Checking remote version for 4.0

Checking if switch is BACKUP for any virtual routers and if

the state is 'No Service'.

Checking vip redundancy state ...

WARNING: This switch is currently the Backup for

a configured virtual router.

Once the script is run this switch will become Master. Unless preempted, it will

remain Master for each and every virtual router configured.

Do you wish to continue [y/n]? y

Working ...

Checking if backup switch is Master for any VRIDs.

If it is, either a local interface that once held redundant vips

has been removed or the Backup is a Master for another vip-redundant

relationship.

Checking remote redundant vips ...

Saving Master running-config to startup-config and archiving startup-config.

Copying running-config to startup-config.

Archiving startup-config.

Swapping Master and Backup ip addresses in tmp.cfg for app

and redundancy interface.

Checking for multiple APP sessions between redundant peers.

App Session IP: 172.17.199.5, Local IP address: 172.17.199.4

Checking IP Address size differences : <172.17.199.5> <172.17.199.4>

Adding APP session IP length difference to LOCALCONFIG byte size : 0

Adding APP session IP length difference to REMOTECONFIG byte size : 0

Removing CIRCUIT and INTERFACE modes from tmp.cfg.

Checking for SSL configuration ...

Working ...

Using rcmd to copy tmp.cfg to a file on Backup switch.

Archiving copy to Backup's startup-config.

Archiving Backup's current startup-config.

Restoring startup-config (new copy) to startup-config.

Clearing running-config.

Script playing the copy script of the Master's running-config.

Checking to make sure backup App goes down

Making sure app session is down.

Seconds to wait before calling it quits: 12629

Copy success being verified by comparing byte sizes of archived running-

configs of the Master switch and the Backup switch.

Making sure app session is up.

Seconds to wait before calling it quits: 12194

Stand By Switch's app session is not up.

Check for script commit_failure on the Backup

Time start 09:11

nmfoxton · ‎05-23-2012

So after 4 hours the result is a failure anf half the configuration plus the app session is missing on the backup CSS.

I'll have to arrange a reboot and try again.

nmfoxton · ‎05-24-2012

Problem solved.

Someone had configured one of the virtual-routers on and abd b side with the same priority and preempt.

This seesm to upset the commit script, once it was removed the replication worked perfectly.

Result

Jorge Bejarano · ‎05-24-2012

Great !!!