Cannot create any port profiles on n1k from VSM

[gdragatsis]

"Totally stumped on this one. Here's the situation:

• Have VSM's up running and connected to vCenter (redundant mode)
  
• Have all VEM's up and running
  
• Have created all required prot profiles, uplinks etc etc.

All of a sudden, trying to create additional prot profiles and I am getting these error messages:

"Warning: Operation succeeded locally but update failed on vCenter server" when trying to create a port profile.

ALso here is the debug logs:

2010 Jan 18 15:10:37.239192 portprofile: ppm_profile_fsm_ac_add_port_group(4162): Profile state is enabled, handshaking with MSP

2010 Jan 18 15:10:37.239245 portprofile: ppm_send_create_pg_req(1110): Payload ready

2010 Jan 18 15:10:37.239310 portprofile: ppm_send_create_pg_req(1122): TLV ready

2010 Jan 18 15:10:37.239383 portprofile: ppm_send_request(61): Sending MTS_OPC_MSP_CREATE_PP to sap 444

2010 Jan 18 15:10:37.263340 portprofile: ppm_mts_handler_init(108): Handling msg MTS_OPC_MSP_CREATE_PP (msg id 23392025)

2010 Jan 18 15:10:37.263442 portprofile: ppm_handle_port_group_create(2231): Handling a mts msg with opcode [MTS_OPC_MSP_CREATE_PP]

2010 Jan 18 15:10:37.263528 portprofile: ppm_timer_stop(95): timer stopped: rid(port-profile/18), event_id(17), timer_id(34)

2010 Jan 18 15:10:37.263585 portprofile: ppm_handle_port_group_create(2264): Status in the response message is :Syserr code: 0x41b10010,     user string: [[VMware vCenter Server 4.0.0 build-208111] ]     internal string: [SOAP 1.1 fault: "":ServerFaultCode [VMware vCenter Server 4.0.0 build-208111] ]

2010 Jan 18 15:10:37.263646 portprofile: ppm_handle_port_group_create(2268): Port group create failure for port-profile/18

2010 Jan 18 15:10:37.263699 portprofile: ppm_handle_port_group_create(2271): MSP Handshake failed.

Warning: Operation succeeded locally but update failed on vCenter server.

Port profile 'xxxxxxxxxxxx' not found.

Has anyone seen this error message?

[Robert Burns]

Looks like a communication issue between your VSM and vCenter.

Can you post the output of (or attach output file if lenghty):

1. show svs connection

2. show vms internal errors

3. show vmware vc extension key

4.  Open up http://[vcenter-ip]/mob/?moid=ExtensionManager and list all the Cisco_Nexus_1000v_xxxx extension keys

The extension key should match with your VSM extension key.  If not there's an issue.

Questions:

Have you changed any of the SSL certificates on the 1000v or your vCenter server by chance?

Have you recently performed a VSM switchoveer?

What version of 1000v are you running?

Robert

[gdragatsis]

sh svs connections

connection VC:

    ip address: xx.xx.xx.xx                

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: xxxxxxxxxxx

    DVS uuid: 3a b0 25 50 5a 3f 0b 6f-0f ee 4f 2d 20 1d 96 28

    config status: Enabled

    operational status: Connected

    sync status: Complete

    version: VMware vCenter Server 4.0.0 build-208111

---------------------------------

SVS domain config:

  Domain id:    1

  Control vlan: 1007

  Packet vlan:  1008

  L2/L3 Control mode: L2

  L3 control interface:  NA

  Status: Config push to VC successful.

--------------------------------------

Questions:

Have you changed any of the SSL certificates on the 1000v or your vCenter server by chance? Nope

Have you recently performed a VSM switchoveer? Nope

What version of 1000v are you running? Nexus1000v.4.0.4.SV1.2

--------------------------------------

So i looked at every possible log from ESXi to vCenter and no luck. even looked at the tomcat logs on vCenter. No luck.

Then I rebooted vCenter. No luck

So eventually rebooted the VSM.

Finally all worked after that. So now i'm looking at the root cause. Can't have it happening and then rebooting all the time. What is this microsoft?

gd

[Robert Burns]

Did you verify if the extension keys matched as per my previous instructions?

By the SOAP error, that could be an issue if the keys were out of sync.

Can you also attach a show tech log

show tech-support svs > bootflash:show_tech.log

Robert