Issue - Can't do a sh run on the VSM's

Timothy Graham · ‎05-04-2011

Hi

I have a problem with a 2 VSM's in to differant VMWare enviroments where by I can not see the configuration, either Sh run or Sh start. All I get back is the below output on the VSM console.

GFD1000VSM-1# sh run

2011 May 3 16:45:09 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 3 16:45:09 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 3 16:45:10 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 3 16:45:10 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 3 16:45:11 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 3 16:45:11 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

Ascii cfg send notify failed: Device or resource busy

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_acquire_q_space() failing - no space in sap 444, uuid 583 send_opc 7663, pid 22657, proc_name vsh - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=3385 rq=0 lq=0 pq=0 nq=0 sq=0 buf_in_transit=20, bytes_in_transit=2316080 - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=403 rq=0 lq=0 pq=0 nq=0 sq=0 buf_in_transit=10226, bytes_in_transit=2089040 - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=445 rq=0 lq=0 pq=21342 nq=0 sq=0 buf_in_transit=0, bytes_in_transit=0 - kernel

2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=444 rq=0 lq=0 pq=10226 nq=0 sq=0 buf_in_transit=21342, bytes_in_transit=4384048 - kernel

GFD1000VSM-1# 2011 May 3 16:45:12 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_deliver_local_atomic:mts_acquire_q_space failed for opcode 7663, src_sap = 11578 - kernel

GFD1000VSM-1#

Any body seen this before or got a fix for it, ver is 4.0(4)SV1(2), all other comands that I have tried seem to work OK, only issue is to do with see the configuration.

Cheer

Tim

sachthak · ‎05-04-2011

Hi Tim,

Could you please provide the output of the command "show system internal mts buffers details"?

Also are you able to create new portgroups on the VC with this system?

Thanks,

Sachin

Timothy Graham · ‎05-05-2011

Hi Sachin

Here is the output of the command you asked to be run from the VSM

I can add the CFG to the VSM, but when I enter the State Enabled command I get the following and port group is not added to the VC, this just seems to keep repeating now.

GFD1000VSM-1(config-port-prof)# state enabled

2011 May 5 07:12:46 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 5 07:12:46 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 5 07:12:47 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 5 07:12:47 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 5 07:12:48 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 5 07:12:48 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available():1121: regular+fast mesg total = 10226, soft limit = 4096 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_is_q_space_available(): NO SPACE sap=444, sap_opt = 0x1, hdr_opt = 0x0, rq=0, lq=0, pq=10226, nq=0, sq=0, fast: rq=0, lq=0, pq=0, nq=0, sq=0 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_acquire_q_space() failing - no space in sap 444, uuid 583 send_opc 114690, pid 4055, proc_name ppm - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=3385 rq=0 lq=0 pq=0 nq=0 sq=0 buf_in_transit=20, bytes_in_transit=2316080 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=403 rq=0 lq=0 pq=0 nq=0 sq=0 buf_in_transit=10226, bytes_in_transit=2089040 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=445 rq=0 lq=0 pq=21342 nq=0 sq=0 buf_in_transit=0, bytes_in_transit=0 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: node=4 sap=444 rq=0 lq=0 pq=10226 nq=0 sq=0 buf_in_transit=21342, bytes_in_transit=4384048 - kernel

2011 May 5 07:12:49 GFD1000VSM-1 %KERN-2-SYSTEM_MSG: mts_deliver_local_atomic:mts_acquire_q_space failed for opcode 114690, src_sap = 429 - kernel

Note: Processing command..

Also noticed on the Standby VSM that there is meesage on login to it that say's who: write error: No space left on device, do not see this message on the active VSM.

Just also tried to TFTP the CFG of the VSM and just get a message int he file it sends say ' Ascii cfg send notify failed: Davie or resource busy.

Neither N1K's are looking good currently, but at least they are working ie pasing traffic to and from the Vmware enviroment at the moment.....

Cheer

Tim

Timothy Graham · ‎05-09-2011

Hi Sachin

Quick update, re booting the VSM with the above error makes the VSM unbootable, thus a rebild of thr VSM is required, On reboot it swapped over to the standby VSM and I was able to do a Sh run on that VSM, not long after swapping it over I got a warning about logs being almost full. So I cleared the logs out and the message did not come back.

So look like the memory is filling up and causin ghte issue to happen, I ned to try clearing the logs on the other N1K we have in use to see if that fixes the issue, will post update after that.

Cheers

Tim

sachthak · ‎05-09-2011

Hi Tim,

Based on the output, it looks like you could be hitting CSCth01542 - "SVS connection service is busy" always returned

Further detail on this bug are available at:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth01542

This was fixed in SV1(3b) or later.

Hope this helps:

Thanks,

Sachin

Timothy Graham · ‎05-10-2011

Hi Sachin,

Thanks for the info, that may explain another issue I had while upgrading the switch over the weekend just gone. In the end I had to bin the existing version and install the latest version as a new VSM/VEM install etc. It was all fun and games.

This has left me with another issue/problem, how do I now license the N1K I have installed. The old one was fully Licensed but as I had to blow away the VSM's and reinstall then license file now do not work, I had a quick look at the licensing web page but could not see a way of removing old now unused license's to allow me to create a new license file to install on the newly build VSM.

Any points would be helpful

Thanks

Tim

sachthak · ‎05-10-2011

Hey Tim,

I've contacted our licensing team to confirm the process and will get back to you shortly.

Cheers,

Sachin

Timothy Graham · ‎06-20-2011

Hi Sachin,

Did you ever get an answer from the licensing team on this at all.....

Cheers

Tim

gghayur · ‎06-20-2011

Hi Tim,

Here is the process for re-installing the license if you have replaced a VSM

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0/license/configuration/guide/license_2install_cfg.html#wp1051852

Timothy Graham · ‎06-21-2011

Thanks.