Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2686
Views
0
Helpful
5
Replies

IVR zoneset activation stuck in SAN-OS 3.3

phaddad
Level 1
Level 1

‎05-04-2012 05:49 PM

After trying to activate a zoneset we keep seeing the following log messages endlessly repeating:

2012 May  4 02:01:45 mds209017901-f1 %ZONE-2-ZS_CHANGE_ACA_FAILED: %$VSAN 265%$ ACA failed : domain 0x63 returns FABRIC_CHANG

ING

2012 May  4 02:01:45 mds209017901-f1 %ZONE-2-ZS_CHANGE_ACTIVATION_FAILED: %$VSAN 265%$ Activation failed

2012 May  4 02:01:45 mds209017901-f1 %ZONE-2-ZS_CHANGE_ACTIVATION_FAILED_RESN_DOM: %$VSAN 265%$ Activation failed : reason Fa

bric changing domain 99

2012 May  4 02:01:45 mds209017901-f1 %IVR-5-IVZS_ACTIVATION_RETRYING: Inter-VSAN zoneset activation did not go through becaus

e of Fabric changing, retrying in VSAN 265 after 15 seconds

2012 May  4 02:01:45 mds209017901-f1 %IVR-2-IVZS_ACTIVATION_FAILED_RETRYING: Inter-VSAN zoneset activation failed in VSAN 265

: Fabric changing.retrying after 15 seconds

also:

sh ivr internal distribution

Distribution internal information

-----------------------------

lock state 1

locked by  admin 00:00:00:00:00:00:00:00

opened : TRUE

activate_or_deactivate : activation

force_activate : FALSE

initiator : TRUE

implicit_commit : FALSE

topology activate: FALSE

msghandle : 1456850992

status : 0

zoneset_name : ivr_zoneset-br2

Ditrib fsm  Current State : IVR_ZONE_DISTRIB_ST_ACTIVATING

   Number of zoneset in pending database : 4

   Number of zones in pending database   : 410

   Number of members in pending database : 2446 sh ivr internal distribution
Distribution internal information
-----------------------------
lock state 1
locked by  admin 00:00:00:00:00:00:00:00
opened : TRUE
activate_or_deactivate : activation
force_activate : FALSE
initiator : TRUE
implicit_commit : FALSE
topology activate: FALSE
msghandle : 1456850992
status : 0
zoneset_name : ivr_zoneset-br2
Ditrib fsm  Current State : IVR_ZONE_DISTRIB_ST_ACTIVATING
   Number of zoneset in pending database : 4
   Number of zones in pending database   : 410
   Number of members in pending database : 2446

I am not at all an expert with SAN, but have been asked to help if possible.  Can anyone provide any way to unwind this stuck activation?  Not sure if clearing the ivr session will help or not and don't want to try it without knowing if it will cause the situation to get worse.  Any suggestions are appreciated.

thanks

P

Labels:
0 Helpful
5 Replies 5

maminhas
Level 1
Level 1

‎05-04-2012 06:10 PM

IVR is complicated , before any changes can be recommended.

with this limited view of error message just from one switch,

you need to understand..

-how many switches are in Fabric

-how many are IVR enabled

-Is it ivr 1 or 2

-what version running on switches..

-what is the ivr topology (show ivr vsan-topology)

It could be a stale domain issue.

find the switch which has domain 0x63  and run show zone internal vsan 265  command on that switch and this switch..

to get a view of if there is any potential stale domain.

typically for IVR issues , you might better be ending with opening a TAC case for extensive diagnosis of show techs on all switches.

0 Helpful

phaddad
Level 1
Level 1
In response to maminhas

‎05-07-2012 10:32 AM

I am not sure how to gather all the requested info regarding # switches in fabric, ivr version etc. I can include the output of show ivr tech if that would be helpful. I "believe" that there are only 2 switches actively participating in the IVR.

mds209817515-f1# show zone internal vsan 265

VSAN: 265 default-zone: deny(rw) distribute: active only

    E_D_TOV: 2000 R_A_TOV: 10000 D_S_TOV: 5000 F_S_TOV: 5000 F_D_TOV: 2000

    Interop: default IOD: disable bcast: disable dflt-bcast: disable dflt-qos: 0

    DBLock:-(F count:0) Ifindex Table Size: 221 Transit Frame Index: 0

    Total Transit Frame Count: 154 Transit Discard Count: 11

Full Database Counters :

    Zonesets: 1 Zones: 1 Huge id zones: 0

    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0

    Aliases: 0 Attribute-groups: 0

    Members: 4 LUN Members: 0 DDAS Members: 0

    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0

Session Database Counters (diff) :

    Zonesets: 0 Zones: 0

    Aliases: 0 Attribute-groups: 0

    Members: 0 LUN Members: 0 DDAS Members: 0

Active Database Counters :

    Zonesets: 1 Zones: 403 Huge id zones: 0

    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0

    Members: 421 LUN Members: 0 DDAS Members: 0

    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0

Session Active Database Counters (diff) :

    Zones: 0 Members: 0 LUN Members: 0 DDAS Members: 0

License Info: 0x0

Full Zoning Database :

    DB size: 196 bytes

    Zonesets:1  Zones:1 Aliases: 0

Active Zoning Database :

    DB size: 41812 bytes

    Name: production_storage  Zonesets:1  Zones:403

TCAM Info :

    cur_seq_num : 184201,  state : 0, def_enum : 0

    gen state : 0,  cur_num_dids : 0 total_num_dids : 0

    num tcam q : 0 num pending q : 0

Change protocol info :

    local domain id = 99,   ACA by 0xff

    State =       Idle,  reply_cnt = 0, req_sent_cnt = 0, req_pending = 0

    Remote domains :

        6(0x6) [21:09:00:0d:ec:11:c3:01]

        23(0x17)

        186(0xba)

        96(0x60)

        150(0x96)

        97(0x61)

        138(0x8a)

        82(0x52)

        16(0x10)

        161(0xa1)

        129(0x81)

        116(0x74)

        20(0x14)

        11(0xb)

        27(0x1b)

        111(0x6f)

        36(0x24)

        2(0x2)

        117(0x75)

        73(0x49)

        102(0x66)

        136(0x88)

        10(0xa)

        223(0xdf)

        108(0x6c)

        103(0x67)

        15(0xf)

        55(0x37)

        26(0x1a)

        91(0x5b)

    Status : Domains mismatch, ACA from dom 6 Rejected!

Merge proto info :

    i/f port-channel 1

       State = Idle       | notify = 0x7 | - -

thanks,

Peter

0 Helpful

maminhas
Level 1
Level 1
In response to phaddad

‎05-08-2012 08:38 PM

Go to switch which has domain 6

on that switch run the same command.

show zone internal vsan 265

that switch has domain mismatch.

0 Helpful

phaddad
Level 1
Level 1
In response to phaddad

‎05-09-2012 06:53 AM

OK here are the outputs from "show zone internal vsan 265" from both switches (domain 6 & domain 99)

domain6:

mds209017901-f1#  show zone internal vsan 265

VSAN: 265 default-zone: deny(rw) distribute: full

    E_D_TOV: 2000 R_A_TOV: 10000 D_S_TOV: 5000 F_S_TOV: 5000 F_D_TOV: 2000

    Interop: default IOD: disable bcast: disable dflt-bcast: disable dflt-qos: 0

    DBLock:-(F count:0) Ifindex Table Size: 230 Transit Frame Index: 0

    Total Transit Frame Count: 74 Transit Discard Count: 0

Full Database Counters :

    Zonesets: 1 Zones: 1 Huge id zones: 0

    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0

    Aliases: 0 Attribute-groups: 0

    Members: 4 LUN Members: 0 DDAS Members: 0

    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0

Session Database Counters (diff) :

    Zonesets: 0 Zones: 0

    Aliases: 0 Attribute-groups: 0

    Members: 0 LUN Members: 0 DDAS Members: 0

Active Database Counters :

    Zonesets: 1 Zones: 403 Huge id zones: 0

    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0

    Members: 421 LUN Members: 0 DDAS Members: 0

    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0

Session Active Database Counters (diff) :

    Zones: 0 Members: 0 LUN Members: 0 DDAS Members: 0

License Info: 0x0

Full Zoning Database :

    DB size: 196 bytes

    Zonesets:1  Zones:1 Aliases: 0

Active Zoning Database :

    DB size: 41812 bytes

    Name: production_storage  Zonesets:1  Zones:403

TCAM Info :

    cur_seq_num : 19742,  state : 0, def_enum : 0

    gen state : 0,  cur_num_dids : 0 total_num_dids : 0

    num tcam q : 0 num pending q : 0

Change protocol info :

    local domain id = 6,   ACA by 0xff

    State =       Idle,  reply_cnt = 1, req_sent_cnt = 1, req_pending = 0

    Remote domains :

        2(0x2)

        10(0xa)

        11(0xb)

        16(0x10)

        20(0x14)

        23(0x17)

        27(0x1b)

        36(0x24)

        73(0x49)

        82(0x52)

        96(0x60)

        97(0x61)

        99(0x63) [21:09:00:0d:ec:3f:37:41]

        102(0x66)

        103(0x67)

        108(0x6c)

        111(0x6f)

        116(0x74)

        117(0x75)

        129(0x81)

        136(0x88)

        138(0x8a)

        150(0x96)

        161(0xa1)

        186(0xba)

        223(0xdf)

        228(0xe4)

        15(0xf)

        55(0x37)

        26(0x1a)

        128(0x80)

        91(0x5b)

    Status : Received RJT from dom 99, RCA Sent!

Merge proto info :

    i/f port-channel 5

       State = Idle       | notify = 0x7 | - -

domain 99:

mds209817515-f1# show zone internal vsan 265
VSAN: 265 default-zone: deny(rw) distribute: active only
    E_D_TOV: 2000 R_A_TOV: 10000 D_S_TOV: 5000 F_S_TOV: 5000 F_D_TOV: 2000
    Interop: default IOD: disable bcast: disable dflt-bcast: disable dflt-qos: 0
    DBLock:-(F count:0) Ifindex Table Size: 230 Transit Frame Index: 0
    Total Transit Frame Count: 154 Transit Discard Count: 11
Full Database Counters :
    Zonesets: 1 Zones: 1 Huge id zones: 0
    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0
    Aliases: 0 Attribute-groups: 0
    Members: 4 LUN Members: 0 DDAS Members: 0
    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0
Session Database Counters (diff) :
    Zonesets: 0 Zones: 0
    Aliases: 0 Attribute-groups: 0
    Members: 0 LUN Members: 0 DDAS Members: 0
Active Database Counters :
    Zonesets: 1 Zones: 403 Huge id zones: 0
    Read-only Zones: 0 QoS Zones: 0 Broadcast Zones: 0
    Members: 421 LUN Members: 0 DDAS Members: 0
    Adv Zoning3 Members(IPv4 + dom-If): 0 IPv6 Members: 0
Session Active Database Counters (diff) :
    Zones: 0 Members: 0 LUN Members: 0 DDAS Members: 0
License Info: 0x0
Full Zoning Database :
    DB size: 196 bytes
    Zonesets:1  Zones:1 Aliases: 0
Active Zoning Database :
    DB size: 41812 bytes
    Name: production_storage  Zonesets:1  Zones:403
TCAM Info :
    cur_seq_num : 184366,  state : 0, def_enum : 0
    gen state : 0,  cur_num_dids : 0 total_num_dids : 0
    num tcam q : 0 num pending q : 0
Change protocol info :
    local domain id = 99,   ACA by 0xff
    State =       Idle,  reply_cnt = 0, req_sent_cnt = 0, req_pending = 0
    Remote domains :
        6(0x6) [21:09:00:0d:ec:11:c3:01]
        23(0x17)
        186(0xba)
        96(0x60)
        150(0x96)
        97(0x61)
        138(0x8a)
        82(0x52)
        16(0x10)
        161(0xa1)
        129(0x81)
        116(0x74)
        20(0x14)
        11(0xb)
        27(0x1b)
        111(0x6f)
        36(0x24)
        2(0x2)
        117(0x75)
        73(0x49)
        102(0x66)
        136(0x88)
        10(0xa)
        223(0xdf)
        108(0x6c)
        103(0x67)
        15(0xf)
        55(0x37)
        26(0x1a)
        128(0x80)
        91(0x5b)
    Status : Domains mismatch, ACA from dom 6 Rejected!
Merge proto info :
    i/f port-channel 1
       State = Idle       | notify = 0x7 | - -

thanks,

peter

0 Helpful

maminhas
Level 1
Level 1
In response to phaddad

‎05-09-2012 09:36 AM

There is domain 228(0xe4)  stale domain.

you need to check all the switches in fabric if  domain 228 is indeed not in use anywhere. (either virtual or non-virtual ).

once confirmed. this should be cleared.

IVR issues are complex in nature, lot of other commands are required to cross verify the findings to eventually confirm stale domain finding or it is something name server not populating a domain .

Like mentioned before opening a TAC case is required to handle this type of issue.

0 Helpful
Customers Also Viewed These Support Documents

Review Cisco Networking for a $25 gift card