Admin context: %ACE-4-106023 denied by #FT_VLAN_ACL#4#????

brian_nc · ‎01-17-2011

I have two ACE's. On the primary ACE, on the Admin context, I am getting the following message on the console:

Jan 17 2011 18:14:12 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

This is the FT VLAN (VLAN 999). The primary, where this message is appearing, is 10.XX.XX.10, and the secondary is 10.XX.XX.11.

The secondary ACE is updating its config when I make any changes on the Primary. I even restarted the primary and everything failed over to the secondary as it should. I assumed the UDP that the ACE is attemping to send out is in regards to the heartbeat. Will this denied message effect failover at all? How can I resolve this issue?

Thanks,

-b

Dan-Ciprian Cicioiu · ‎01-17-2011

Hi,

No.

The FT VLAN is a dedicated interface that the ACE uses exclusively for redundancy traffic (such as heartbeat, state, and replication packets).

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA1_7_/command/reference/ftintfce.html

Dan

brian_nc · ‎01-17-2011

Dan, thanks for the reply. The FT_VLAN interface is in a dedicated VLAN (999). Only the primary and secondary ACE's are in it.

With that said, why is the primary ACE sending out UDP packets to the secondary ACE, only to have them blocked (it appears by the message). There are several hundred of these per minute. I have copied below the six messages on the console that appeared during the span of 1 second.

It appears by these messages that an access-group "#FT_VLAN_ACL#4#" is blocking the attempt by the primary ACE (10.XX.XX.10) to send a UDP message to the secondary ACE (10.XX.XX.11) over the FT_VLAN (VLAN999).

There is no such access-group. No access-groups are applied globaly. A policy map is applied to the management interface which is called "REMOTE_ACCESS".

When I reload the primary ACE, everything fails over to the secondary ACE which then starts sending out these blocked (apparently) UDP messages to the primary ACE.

Thoughts?

Jan 17 2011 18:14:11 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

m Jan 17 2011 18:14:11 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

Jan 17 2011 18:14:11 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

no Jan 17 2011 18:14:11 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

Jan 17 2011 18:14:11 : %ACE-4-106023: Deny udp src vlan999:10.XX.XX.10/50002 dst undetermined:10.XX.XX.11/50000 by access-group "#FT_VLAN_ACL#4#" [0xffffffff, 0x0]

Thanks,

-b

Dan-Ciprian Cicioiu · ‎01-17-2011

I would look at netmask/config of the vlan 999 , or FT config
even though you have the source ok , then "dst undetermined:10.XX.XX.11/50000".

Can you post the FT config ?

Dan

brian_nc · ‎01-17-2011

Here is the FT config of the primary ACE:

interface port-channel 1
ft-port vlan 999
switchport trunk allowed vlan 7,15,40,741,744-745,748
port-channel load-balance src-dst-port
no shutdown

ft interface vlan 999
ip address 10.9.99.10 255.255.255.0
peer ip address 10.9.99.11 255.255.255.0
no shutdown

ft peer 1
heartbeat interval 300
heartbeat count 20
ft-interface vlan 999
query-interface vlan 7
ft group 1
peer 1
priority 250
peer priority 200
associate-context Admin
inservice

Here is the FT config of the secondary ACE:

interface port-channel 1
ft-port vlan 999
switchport trunk allowed vlan 7,15,40,741,744-745,748
port-channel load-balance src-dst-port
no shutdown

ft interface vlan 999
ip address 10.9.99.11 255.255.255.0
peer ip address 10.9.99.10 255.255.255.0
no shutdown

ft peer 1
heartbeat interval 300
heartbeat count 20
ft-interface vlan 999
query-interface vlan 7
ft group 1
peer 1
priority 200
peer priority 250
associate-context Admin
inservice

Dan-Ciprian Cicioiu · ‎01-17-2011

Can you post also :

show ft stats
show ft peer detail
show ft group detail

Dan

brian_nc · ‎01-17-2011

Here they are from the primary:

show ft stats

HA Heartbeat Statistics
------------------------

Number of Heartbeats Sent                 : 28547
Number of Heartbeats Received             : 85181
Number of Heartbeats Missed               : 155
Number of Unidirectional HB's Received    : 11
Number of HB Timeout Mismatches           : 0
Num of Peer Up Events Sent                : 1
Num of Peer Down Events Sent              : 0
Successive HB's miss Intervals counter    : 0
Successive Uni HB's recv counter          : 0

show ft peer detail

Peer Id                      : 1
State                        : FSM_PEER_STATE_COMPATIBLE
Maintenance mode             : MAINT_MODE_OFF
FT Vlan                      : 999
FT Vlan IF State             : UP
My IP Addr                   : 10.9.99.10
Peer IP Addr                 : 10.9.99.11
Query Vlan                   : 7
Query Vlan IF State          : UP
Peer Query IP Addr           : 172.26.7.9
Heartbeat Interval           : 300
Heartbeat Count              : 20
Tx Packets                   : 1771
Tx Bytes                     : 419944
Rx Packets                   : 1750
Rx Bytes                     : 410753
Rx Error Bytes               : 0
Tx Keepalive Packets         : 1692
Rx Keepalive Packets         : 1692
TL_CLOSE count               : 0
FT_VLAN_DOWN count           : 0
PEER_DOWN count              : 0
SRG Compatibility            : COMPATIBLE
License Compatibility        : COMPATIBLE
FT Groups                    : 3

show ft group detail

FT Group                     : 1
No. of Contexts              : 1
Context Name                 : Admin
Context Id                   : 0
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
My Config Priority           : 250
My Net Priority              : 250
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority         : 200
Peer Net Priority            : 200
Peer Preempt                 : Enabled
Peer Id                      : 1
Last State Change time       : Mon Jan 17 17:49:50 2011
Running cfg sync enabled     : Enabled
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync enabled     : Enabled
Startup cfg sync status      : Startup configuration sync has completed
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0

Dan-Ciprian Cicioiu · ‎01-17-2011

In my opinion the bevior isn't a normal one.

Have you try searching the BugToolKit or even open a TAC case ?

Dan

brian_nc · ‎01-17-2011

I've not tried the bugtoolkit. I may end up having to open a TACS case, but decided to start here to see if any of the dev's will be able to assist.

Has anyone else any insight into this error?

Thanks,

-b

brian_nc · ‎01-18-2011

bump

brian_nc · ‎01-21-2011

This error was due to LACP being used as the channel-protocol on the 6506 that had all 4 ACE ports plugged in to it.

I removed all 4 ports from the port-channel, removed the channel-protocol, then added them all back one at a time using "channel-group 1 mode on".

The error message was apparently due to the LACP protocol vs regular ole etherchannel protocol.

Doing a "show etherchannel summary" on the switch that the ACE was plugged in to show that the interfaces were bundling together as they should, even though I was able to telnet to the ACE, and all FT failovers, syncs, etc... continued to work.

-b