ACE standby unit is failing | FT group is not sync

gargolek99 · ‎12-05-2014

Hello,

We had upgrade to A5 3.0 and reload device device has lost all its configuration, after that we downgraded to same version as primary A5 1.2 and add back basic config, once we enable FT groups devices going thre sync process and displays below:

config-ft-group)#

NOTE: Configuration mode is enabled on all sessions

`no interface gigabitEthernet 1/4`
*** Context 0: cmd exec error ***

`no interface gigabitEthernet 1/3`
*** Context 0: cmd exec error ***

`no interface gigabitEthernet 1/2`
*** Context 0: cmd exec error ***

`no interface gigabitEthernet 1/1`
*** Context 0: cmd exec error ***

%ACE-4-411003: Interface gigabitEthernet1/4 changed state to administratively u
p
%ACE-4-411003: Interface vlanxxx changed state to administratively up
%ACE-4-411001: Line protocol on Interface vlanxxx, changed state to up
%ACE-2-727016: HA: Configuration Replication status for context Admin - Startup
configuration has been applied successfully for context.
%ACE-1-727001: HA: Peer 10.10.11.1 is not reachable. Error: Heartbeat stopped.
No alternate interface configured
%ACE-5-111008: User 'admin' executed the 'interface gigabitEthernet 1/1' comman
d.
%ACE-6-615003: VLAN 0 not available for configuring an interface
%ACE-6-615003: VLAN 2 not available for configuring an interface
%ACE-6-615003: VLAN 3 not available for configuring an interface
%ACE-6-615003: VLAN 4 not available for configuring an interface
%ACE-6-615003: VLAN 5 not available for configuring an interface
%ACE-6-615003: VLAN 6 not available for configuring an interface
%ACE-6-615003: VLAN 7 not available for configuring an interface
%ACE-6-615003: VLAN 8 not available for configuring an interface
%ACE-6-615003: VLAN 9 not available for configuring an interface
%ACE-6-615003: VLAN 10 not available for configuring an interface
%ACE-6-615003: VLAN 11 not available for configuring an interface
%ACE-6-615003: VLAN 12 not available for configuring an interface
%ACE-6-615003: VLAN 13 not available for configuring an interface
%ACE-6-615003: VLAN 14 not available for configuring an interface
%ACE-6-615003: VLAN 15 not available for configuring an interface
%ACE-6-615003: VLAN 16 not available for configuring an interface
%ACE-6-615003: VLAN 17 not available for configuring an interface
%ACE-6-615003: VLAN 18 not available for configuring an interface
%ACE-6-615003: VLAN 19 not available for configuring an interface
%ACE-6-615003: VLAN 20 not available for configuring an interface
%ACE-6-615003: VLAN 21 not available for configuring an interface
%ACE-6-615003: VLAN 22 not available for configuring an interface
%ACE-6-615003: VLAN 23 not available for configuring an interface
%ACE-6-615003: VLAN 24 not available for configuring an interface
%ACE-6-615003: VLAN 25 not available for configuring an interface
%ACE-6-615003: VLAN 26 not available for configuring an interface
%ACE-6-615003: VLAN 27 not available for configuring an interface
%ACE-6-615003: VLAN 28 not available for configuring an interface
%ACE-6-615003: VLAN 29 not available for configuring an interface
%ACE-6-615003: VLAN 30 not available for configuring an interface
%ACE-6-615003: VLAN 31 not available for configuring an interface
%ACE-6-615003: VLAN 32 not available for configuring an interface
%ACE-6-615003: VLAN 33 not available for configuring an interface
%ACE-6-615003: VLAN 34 not available for configuring an interface
%ACE-6-615003: VLAN 35 not available for configuring an interface
%ACE-6-615003: VLAN 36 not available for configuring an interface
%ACE-6-615003: VLAN 37 not available for configuring an interface
%ACE-6-615003: VLAN 38 not available for configuring an interface
%ACE-6-615003: VLAN 39 not available for configuring an interface
%ACE-6-615003: VLAN 40 not available for configuring an interface
%ACE-6-615003: VLAN 41 not available for configuring an interface
%ACE-6-615003: VLAN 42 not available for configuring an interface
%ACE-6-615003: VLAN 43 not available for configuring an interface
%ACE-6-615003: VLAN 44 not available for configuring an interface
%ACE-6-615003: VLAN 45 not available for configuring an interface
%ACE-6-615003: VLAN 46 not available for configuring an interface
%ACE-6-615003: VLAN 47 not available for configuring an interface
%ACE-6-615003: VLAN 48 not available for configuring an interface
%ACE-6-615003: VLAN 49 not available for configuring an interface
%ACE-6-615003: VLAN 50 not available for configuring an interface
%ACE-6-615003: VLAN 51 not available for configuring an interface
%ACE-6-615003: VLAN 52 not available for configuring an interface
%ACE-6-615003: VLAN 53 not available for configuring an interface
%ACE-6-615003: VLAN 54 not available for configuring an interface
%ACE-6-615003: VLAN 55 not available for configuring an interface
%ACE-6-615003: VLAN 56 not available for configuring an interface
%ACE-6-615003: VLAN 57 not available for configuring an interface
%ACE-6-615003: VLAN 58 not available for configuring an interface
%ACE-6-615003: VLAN 59 not available for configuring an interface
%ACE-6-615003: VLAN 60 not available for configuring an interface
%ACE-6-615003: VLAN 61 not available for configuring an interface
%ACE-6-615003: VLAN 62 not available for configuring an interface
%ACE-6-615003: VLAN 63 not available for configuring an interface
%ACE-6-615003: VLAN 64 not available for configuring an interface
%ACE-6-615003: VLAN 65 not available for configuring an interface
%ACE-6-615003: VLAN 66 not available for configuring an interface
%ACE-6-615003: VLAN 67 not available for configuring an interface
%ACE-6-615003: VLAN 68 not available for configuring an interface
%ACE-6-615003: VLAN 69 not available for configuring an interface
%ACE-6-615003: VLAN 70 not available for configuring an interface
%ACE-6-615003: VLAN 71 not available for configuring an interface
%ACE-6-615003: VLAN 72 not available for configuring an interface
%ACE-6-615003: VLAN 73 not available for configuring an interface
%ACE-6-615003: VLAN 74 not available for configuring an interface
%ACE-6-615003: VLAN 75 not available for configuring an interface
%ACE-6-615003: VLAN 76 not available for configuring an interface
%ACE-6-615003: VLAN 77 not available for configuring an interface
%ACE-6-615003: VLAN 78 not available for configuring an interface
%ACE-6-615003: VLAN 79 not available for configuring an interface
%ACE-6-615003: VLAN 80 not available for configuring an interface
%ACE-6-615003: VLAN 81 not available for configuring an interface
%ACE-6-615003: VLAN 107 not available for configuring an interface
%ACE-6-615003: VLAN 134 not available for configuring an interface
%ACE-6-615003: VLAN 161 not available for configuring an interface
%ACE-6-615003: VLAN 188 not available for configuring an interface
%ACE-6-615003: VLAN 215 not available for configuring an interface
%ACE-6-615003: VLAN 242 not available for configuring an interface
%ACE-6-615003: VLAN 266 not available for configuring an interface
%ACE-6-615003: VLAN 292 not available for configuring an interface
%ACE-6-615003: VLAN 319 not available for configuring an interface
%ACE-6-615003: VLAN 346 not available for configuring an interface
%ACE-6-615003: VLAN 373 not available for configuring an interface
%ACE-6-615003: VLAN 400 not available for configuring an interface
%ACE-6-615003: VLAN 427 not available for configuring an interface
%ACE-6-615003: VLAN 455 not available for configuring an interface
%ACE-6-615003: VLAN 482 not available for configuring an interface
%ACE-6-615003: VLAN 508 not available for configuring an interface
%ACE-6-615003: VLAN 534 not available for configuring an interface
%ACE-6-615003: VLAN 561 not available for configuring an interface
%ACE-6-615003: VLAN 589 not available for configuring an interface
%ACE-6-615003: VLAN 616 not available for configuring an interface
%ACE-6-615003: VLAN 643 not available for configuring an interface
%ACE-6-615003: VLAN 668 not available for configuring an interface
%ACE-6-615003: VLAN 695 not available for configuring an interface
%ACE-6-615003: VLAN 749 not available for configuring an interface
%ACE-6-615003: VLAN 775 not available for configuring an interface
%ACE-6-615003: VLAN 802 not available for configuring an interface
%ACE-6-615003: VLAN 829 not available for configuring an interface
%ACE-6-615003: VLAN 856 not available for configuring an interface
%ACE-6-615003: VLAN 884 not available for configuring an interface
%ACE-6-615003: VLAN 903 not available for configuring an interface
%ACE-6-615003: VLAN 930 not available for configuring an interface
%ACE-6-615003: VLAN 957 not available for configuring an interface
%ACE-6-615003: VLAN 983 not available for configuring an interface
%ACE-5-111008: User 'admin' executed the 'channel-group 1' command.
%ACE-4-411003: Interface gigabitEthernet1/1 changed state to administratively u
p
%ACE-5-111008: User 'admin' executed the 'no shutdown' command.
%ACE-5-111008: User 'admin' executed the 'interface gigabitEthernet 1/2' comman
d.
%ACE-5-111008: User 'admin' executed the 'channel-group 1' command.
%ACE-4-411003: Interface gigabitEthernet1/2 changed state to administratively u
p
%ACE-5-111008: User 'admin' executed the 'no shutdown' command.
%ACE-5-111008: User 'admin' executed the 'interface gigabitEthernet 1/3' comman
d.
%ACE-5-111008: User 'admin' executed the 'description failover' command.
%ACE-5-111008: User 'admin' executed the 'ft-port vlan xxx' command.
%ACE-4-411003: Interface gigabitEthernet1/3 changed state to administratively u
p
%ACE-4-411001: Line protocol on Interface gigabitEthernet1/3, changed state to
up
%ACE-4-411001: Line protocol on Interface vlanxxx, changed state to up
%ACE-4-411001: Line protocol on Interface gigabitEthernet1/1, changed state to
up
%ACE-4-411001: Line protocol on Interface gigabitEthernet1/2, changed state to
up
%ACE-5-111008: User 'admin' executed the 'no shutdown' command.
%ACE-5-111008: User 'admin' executed the 'interface gigabitEthernet 1/4' comman
d.
%ACE-4-411004: Interface gigabitEthernet1/4 changed state to administratively d
own
%ACE-4-411002: Line protocol on Interface gigabitEthernet1/4, changed state to
down
%ACE-5-111008: User 'admin' executed the 'shutdown' command.
%ACE-1-727001: HA: Peer 10.10.11.1 is not reachable. Error: Heartbeat stopped.
No alternate interface configured
%ACE-5-111008: User 'admin' executed the 'interface port-channel 1' command.
%ACE-6-615004: VLAN 3 available for configuring an interface
%ACE-6-615004: VLAN 105 available for configuring an interface
%ACE-6-615004: VLAN 106 available for configuring an interface
%ACE-6-615004: VLAN 107 available for configuring an interface
%ACE-6-615004: VLAN 115 available for configuring an interface
%ACE-5-111008: User 'admin' executed the 'switchport trunk allowed vlan 3,105-1
07,115' command.
%ACE-4-411003: Interface port-channel1 changed state to administratively up
%ACE-4-411001: Line protocol on Interface port-channel1, changed state to up
%ACE-5-111008: User 'admin' executed the 'no shutdown' command.
%ACE-6-302022: Built TCP connection 0x1c for vlanxx:x,.x.x.x.x/47336 (x.x.x.x.x/47336)

to vlanxxx:x.x.x.x/2000 (x.x.x.x.x/2000)
%ACE-2-727013: HA: Peer 1 is UP and reachable.
%ACE-5-727021: HA: Peer 1 is compatible. Software: COMPATIBLE, License: COMPATI
BLE
%ACE-5-111008: User 'admin' executed the 'ft peer 1' command.
%ACE-5-111008: User 'admin' executed the 'heartbeat interval 300' command.
%ACE-5-111008: User 'admin' executed the 'heartbeat count 10' command.
%ACE-5-111008: User 'admin' executed the 'ft-interface vlan xxx' command.
%ACE-5-111008: User 'admin' executed the 'ft group 1' command.
%ACE-5-111008: User 'admin' executed the 'peer 1' command.
%ACE-5-111008: User 'admin' executed the 'no preempt' command.
%ACE-5-111008: User 'admin' executed the 'associate-context Admin' command.
%ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_ELECT. Event: FSM_F
T_EV_ENABLE_INSERVICE
%ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_ELECT. Event: FSM_F
T_EV_ENABLE_INSERVICE
%ACE-5-111008: User 'admin' executed the 'inservice' command.
%ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_STANDBY_CONFIG. Eve
nt: FSM_FT_EV_STATE
%ACE-2-727012: HA: FT Group 1 changed state to FSM_FT_STATE_INIT. Event: FSM_FT
_EV_DISABLE_INSERVICE
%ACE-4-411004: Interface port-channel1 changed state to administratively down
%ACE-4-411002: Line protocol on Interface port-channel1, changed state to down
%ACE-6-615003: VLAN 3 not available for configuring an interface
%ACE-6-615003: VLAN 105 not available for configuring an interface
%ACE-6-615003: VLAN 106 not available for configuring an interface
%ACE-6-615003: VLAN 107 not available for configuring an interface
%ACE-6-615003: VLAN 115 not available for configuring an interface
%ACE-4-411001: Line protocol on Interface port-channel1, changed state to up
%ACE-4-411002: Line protocol on Interface port-channel1, changed state to down
%ACE-4-411003: Interface gigabitEthernet1/4 changed state to administratively u
p
%ACE-4-411004: Interface gigabitEthernet1/3 changed state to administratively d
own
%ACE-4-411002: Line protocol on Interface gigabitEthernet1/3, changed state to
down
%ACE-1-727001: HA: Peer x.x.x.x is not reachable. Error: Heartbeat stopped.
No alternate interface configured
%ACE-4-411002: Line protocol on Interface vlanxxx, changed state to down
%ACE-6-615003: VLAN xxx not available for configuring an interface
%ACE-4-411004: Interface gigabitEthernet1/2 changed state to administratively d
own
%ACE-4-411002: Line protocol on Interface gigabitEthernet1/2, changed state to
down

Also i had notice that on Primary device during the sync i saw message:

who: write error: No space left on device

I checked the space and looks ok....i am lost now.....need help please.

Regards,

Gargolek

Kanwaljeet Singh · ‎12-06-2014

Hi Gargolek,

If you are getting this message "No space left on device", kindly immediate contact TAC and they will provide you with workaround script to get rid of this issue. PLEASE DO NOT reload the device or effect will be catastrophic. You will lose all the config. Can you check if you have startup-config there on the device or not? I assume it won't be there. Even if it is there, i would suggest to open a TAC case and let them handle this. It is a known issue.

Regards,

Kanwal

Note: Please mark answers if they are helpful.