Nexus 5000 Config-Sync and TACACS problem with the Switch Profile

[greg.murray]

I have configured up config-sync between a pair of Nexus 5020 Switches, and the config-sync element is working quite nicely.

I have the network all set up for TACACS (AAA) and the authentication and authorization is performed by Cisco ACS Appliance 1121 (S/W 5.1). When I log into the switch with my account, which is the account which was used to do all the conifgurations I can access the switch-profile I have created. The problem I do have is that when I access the switch-profile using another TACACS account, I get the following error message

N5K-G37-ACC-01(config-sync)# switch-profile sync
Error: Session is owned by another user

I have created a host of accounts on the ACS and still the problem exists. I have also tested this when I disconnect the ACS and login to the switch using the local account I still get the error.

Software:
  kickstart: version 5.0(2)N1(1)
  system:    version 5.0(2)N1(1)

Any help would be appreciated?

Thanks

Greg

[Jason Aarons]

I have this same error, but I'm not using TACACS but local authentication, I'm on the console and no one else is connected/telnet/ssh. Maybe it's just the switch-profile command can't be entered in config sync mode?

5548a(config-sync)# switch-profile 5548Error: Session is owned by another user

5548a(config-sync)# show switch-profile 5548 status

switch-profile  : 5548----------------------------------------------------------

Start-time: 982000 usecs after Fri Dec  3 22:57:34 2010
End-time: 218159 usecs after Fri Dec  3 22:57:39 2010

Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: No
Profile-status: Sync Success

Local information:
----------------
Status: Commit Success
Error(s):

Peer information:
----------------
IP-address: 10.50.102.62
Sync-status: In Sync.
Status: Commit Success
Error(s):

5548A(config-sync)#

[Matt Craig]

I just recently ran into this issue too, and I believe I know why.  It would appear that the config sync sessions remains active always, and that only the owner of the session (the one who created the session) may access the session.

For example:

I log into the switch with my username 'blarg', and create the switch-profile 'zomg':

switch1# conf sync
switch1(config-sync)# switch-profile zomg
Switch-Profile started, Profile ID is 1
switch1(config-sync-sp)#

I can make changes and sync here no problem.

Then if my collegue John Doe wants to log in an make changes, he logs in with his username 'asdf':

switch1# conf sync
switch1(config-sync)# switch-profile zomg
Error: Session is owned by another user
switch1(config-sync)#
switch1(config-sync)# sh configuration session sum
Session Manager Database:
--------------------------------------------------------------------
Name                    Session Owner           Creation Time
--------------------------------------------------------------------
zomg                    blarg                17:31:57 MST Dec 06 2010

Number of active configuration sessions = 1
switch1(config-sync)#

This, if my understanding is correct, effectively means only one access account may make use of the sync feature.

Lame.

Perhaps I've got a case of PEBKAC and someone can refute, but the docs kind of allude to this.

Refrence http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/system_management/502_n1_1/Cisco_n5k_system_mgmt_cg_rel_502_n1_1_chapter3.html#concept_FEFF536BD8BA42019C859E6C6533E44D

"One switch profile session can be in progress at a time. Attempts to start another session will fail."

[greg.murray]

I've also just found this bug associated with the code.

CSCtj93445    Make switch-profile editable by users with same role

I've been told via Cisco TAC that a new release of code is due out in December 2010 to fix this problem

Thanks

[Abel JULIAN]

Hello,

I have a similar problem and I am running out of ideas.

I have a core of two Nexus 5000 which are running the same version. The second switch has rebooted and cannot merge its conf with switch-profile.

  kickstart: version 5.2(1)N1(1)

  system:    version 5.2(1)N1(1)

There is only one command into the profile but I cannot commit it. (moreover it is a copy run start). I cannot delete the buffer too, it says :

Error: Session Database already locked, Verify/Commit in Progress.

Basically, I am unable to pass any command on the secondary switch (LTS2) and zero switch-profile command on the primary.

Has anyone an idea ?

N5K-LTS1-1# show switch-profile buffer

switch-profile  : BI

----------------------------------------------------------

Seq-no  Command

----------------------------------------------------------

1.1       copy running-config startup-config

N5K-LTS1-1# config sync

Enter configuration commands, one per line.  End with CNTL/Z.

N5K-LTS1-1(config-sync)# switch-profile BI

Switch-Profile started, Profile ID is 1

N5K-LTS1-1(config-sync-sp)# buffe

buffer-delete   buffer-move

N5K-LTS1-1(config-sync-sp)# buffer-delete all

Error: Session Database already locked, Verify/Commit in Progress.

N5K-LTS1-1(config-sync-sp)#

N5K-LTS1-1# sh configuration session sum

Session Manager Database:

--------------------------------------------------------------------

Name                    Session Owner           Creation Time

--------------------------------------------------------------------

BI                      adminlan                17:36:24 WET Nov 20 2012

There are no active configuration sessions

N5K-LTS2-1# show switch-profile status

switch-profile  : BI

----------------------------------------------------------

Start-time: 795782 usecs after Tue Dec 11 10:23:11 2012

End-time: -

Profile-Revision: 1

Session-type: -

Session-subtype: -

Peer-triggered: No

Profile-status: -

Local information:

----------------

Status: -

Error(s):

Peer information:

----------------

IP-address: 172.0.0.0

Sync-status: Not yet merged

Merge Flags: pending_merge:1 rcv_merge:0 pending_validate:0

Status: -

Error(s):

N5K-LTS2-1# sh configuration session sum

Session Manager Database:

--------------------------------------------------------------------

Name                    Session Owner           Creation Time

--------------------------------------------------------------------

BI                      adminlan                09:59:28 WET Dec 05 2012

There are no active configuration sessions

N5K-LTS2-1#

Thanks by advance,

Abel

[greg.murray]

It doesn't look like the switch-profile is properly synced and merged correctly. Its been a while since I have touched config-sync now as I found it too unreliable.

You have only attaced the switch-profile status from one of the N5K's, what does the other one say?

Perhaps delete the switch-profile and start again but it can be quite messy removing it so be careful when doing this.

Regards

Greg

[Abel JULIAN]

I try a lot of things reading the trobloushooting guide. But I was unable to enter any command, I had an error each time from one or another Nexus (Session locked).

I finally opened a Cisco TAC case and Cisco had this issue a very few times. The only workaround they have found to resolv this issue is reboot the switch locking the session. I was a network core for my customer so a reboot was not easy to schedule but I did it.

The problem is solved now but is shows the limitation of config-sync...

Regards,

Abel

[rsjordan00]

I'm having the same issue Abel. I configured a switch profile and it was working for a few days. I then tried to make another change and I got the same error:

Error: Session Database already locked, Verify/Commit in Progress.

I opened a TAC case and we did find that the session manager was locking the session. The process was cleared but we still get the same error. I'm now being told that a reboot is the only fix. We are not in production yet so I can reboot them but it seems unacceptable for an enterprise solution.

[Abel JULIAN]

Hello,

I got the same answer from the TAC and I totally agree with you that a reboot in production must never be a workaround for that kind of equipment.

Despites that I work a litlle bit with Cisco to try to understand the "why" and they were not able to gave me an precise answer. They told me that we are a few with this issue and they cannot reproduce it in lab.

I did not have any other problem since the reboot and I hope It  will not happen again because the Nexus are running in a hospital environment.

Sorry for not helping you more on this, If i have some news I will post them here.

Regards,

Abel

[rsjordan00]

The TAC engineer claimed that it was very rare, but it happened to me after only four days, so I was skeptical. It's good to know that it hasn't reoccurred for you.

Thanks

[HELIANTHUS]

I'm having the same problem now.I do the config-t configration on my one N5K device.and then whatever I do on my device ,It always say "Error: Session Database already locked, Verify/Commit in Progress."

I tried a lot  to fix it ,but it does't work.Now I am contacting The TAC engineer and hope they can give me a solution to solve that problem without reload the switch...It's not rare now ,isn't it ?

[rsjordan00]

I should have posted an update before but I forgot. This issue kept reoccuring on my switches. I reopened my TAC case and they had me run about a dozen different show commands. They still weren't able to pinpoint the problem. At the time, my switches weren't in production yet, so I would simply reload to fix the issue.

I noticed that I never received the session database error during the day when I was doing around 30 config syncs. The following morning, I would see the error during my first config sync. This pointed me towards a nightly backup task run by Solarwinds Orion NCM. All the task did was SSH into each switch and run "copy run start". After I disabled the task I stopped getting the session database errors. I sent NCM logs over to Cisco but they weren't able to see what the issue was. I had to get the switches into production so I didn't have anymore time to test it out.

You can check with TAC to see if there is anything they can do, but chances are you will have to reload the switch. They submitted a bug for my TAC case. It is CSCue03528. The fix is still pending.

[HELIANTHUS]

Thanks for your post...The cisco sales engineer just replied me that this issue in version 5.2(1)N1(1a) is a bug.It aleadly fiexed in the lastest version .5.2x and 6.0 version. You said right the only way to fix it in this version is reboot the switch.But this will make the traffic cuted a few ms. I saw another thing you said that this issue will keep reoccuring ?that is really help me.thank you.

[MARK BAKER]

I had an issue where config-sync would not sync because one switch had the db locked. I issued the command "show system internal csm info trace" to see what was showing it locked. It was whatever command I had entered last. I was able to reset the lock without a reboot with the command "test csm ssn-db-lock reset conf-t". After that config-sync merge succeeded.

Mark

[johnnylingo]

Interesting.  I couldn't get that command on a 5548 running 7.0(8)N1.  Anyway, I just got off the phone with TAC and the only solution they could provide was to reboot both switches in the config sync pair.