cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
cancel
80144
Views
5
Helpful
4
Replies

N1k VSM SVS to vCenter drops off/reconnects intermittently,

ray.deasy
Level 1
Level 1

Hi,

I'm having issues with connectivity from my VSM to vCenter. I have active/standby VSM's. Every so often the connection from the VSM to vCenter drops off & later reconnects. The VEM's stay online and the VSM's stay online. There appears to be connectivity issues to the vCenter itself, i'm thinking it may be ssl/cert related but would appreciate a second opinion/advise.

Here are some log output, if someone could help me out as to what the issue may be it would be great.

Thanks,

Ray.

N1KV-PROD-VSM# sh logging last 30

2013 Feb 11 23:57:01 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.Timeout. SSL connect failed in tcp_connect()

2013 Feb 11 23:57:49 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:00:49 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:02:09 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:05:40 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:06:50 N1KV-PROD-VSM %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin o

n 157.190.4.113@pts/0

2013 Feb 12 00:07:14 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.Timeout. SSL connect failed in tcp_connect()

2013 Feb 12 00:08:12 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:11:12 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:11:43 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.Timeout. SSL connect failed in tcp_connect()

2013 Feb 12 00:13:54 N1KV-PROD-VSM %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully push

ed to the management server.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.Timeout. SSL connect failed in tcp_connect()

2013 Feb 12 00:14:26 N1KV-PROD-VSM %VMS-3-CONNECTION_ERROR: Unable to communicate with vCenter

Server/ESX. Disconnecting..

2013 Feb 12 00:14:26 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'MGMT_Uplink

' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'Unused_Or_Q

uarantine_Uplink' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'VM_Data_Upl

ink' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'V_L3_CONTRO

L_99' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:14:26 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:16:34 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:20:24 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:21:42 N1KV-PROD-VSM %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin o

n 157.190.4.113@pts/0

2013 Feb 12 00:22:49 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.End of file or no input: Operation interrupted or timed out after 160

s send or 160s receive delay.

2013 Feb 12 00:23:40 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:23:40 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:25:49 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VM

ware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed

due to connection failure.End of file or no input: Operation interrupted or timed out after 160

s send or 160s receive delay.

2013 Feb 12 00:25:53 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

2013 Feb 12 00:28:52 N1KV-PROD-VSM %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin o

n 157.190.4.113@pts/0

2013 Feb 12 00:28:53 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the v

Center Server.

2013 Feb 12 00:29:57 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected fr

om the vCenter Server.

N1KV-PROD-VSM#   

N1KV-PROD-VSM#

N1KV-PROD-VSM# sh vms internal event-history errors

1) Event:E_DEBUG, length:53, at 431472 usecs after Tue Feb 12 00:34:05 2013

    [102] waitForTask(5968): Error in creating Task filter

2) Event:E_DEBUG, length:216, at 431466 usecs after Tue Feb 12 00:34:05 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

3) Event:E_DEBUG, length:49, at 431422 usecs after Tue Feb 12 00:34:05 2013

    [102] createFilter(5838): Error in creating filter

4) Event:E_DEBUG, length:103, at 662124 usecs after Tue Feb 12 00:33:33 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

5) Event:E_DEBUG, length:83, at 662117 usecs after Tue Feb 12 00:33:33 2013

    [102] updateDVSBySwitchUUID(3907): failed to retrieve dvs info by uuid. returning -1

6) Event:E_DEBUG, length:216, at 662112 usecs after Tue Feb 12 00:33:33 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

7) Event:E_DEBUG, length:77, at 662090 usecs after Tue Feb 12 00:33:33 2013

    [102] getDVSInfoBySwitchUUID(4200): failed to get object properties. returning

8) Event:E_DEBUG, length:216, at 662076 usecs after Tue Feb 12 00:33:33 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

9) Event:E_DEBUG, length:57, at 662033 usecs after Tue Feb 12 00:33:33 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

10) Event:E_DEBUG, length:81, at 322004 usecs after Tue Feb 12 00:33:00 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

11) Event:E_DEBUG, length:69, at 321922 usecs after Tue Feb 12 00:33:00 2013

    [102] vms_connect(1837): Failed to remove visibility role (0x41b1000c)

12) Event:E_DEBUG, length:106, at 321805 usecs after Tue Feb 12 00:33:00 2013

    [102] vms_vim_delete_visibility_role(3741): Error occurred while searching for visibility r

ole (0x41b1000c)

13) Event:E_DEBUG, length:104, at 321800 usecs after Tue Feb 12 00:33:00 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: (null) [VMware vCenter Server 5.0.0

build-623373]

14) Event:E_DEBUG, length:216, at 342035 usecs after Tue Feb 12 00:29:57 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

15) Event:E_DEBUG, length:60, at 342013 usecs after Tue Feb 12 00:29:57 2013

    [102] waitForTask(6001): ERROR in Retreiving taskInfo from VC

16) Event:E_DEBUG, length:216, at 342005 usecs after Tue Feb 12 00:29:57 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

17) Event:E_DEBUG, length:216, at 341974 usecs after Tue Feb 12 00:29:57 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

18) Event:E_DEBUG, length:57, at 341923 usecs after Tue Feb 12 00:29:57 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

19) Event:E_DEBUG, length:216, at 81730 usecs after Tue Feb 12 00:29:25 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

20) Event:E_DEBUG, length:84, at 81708 usecs after Tue Feb 12 00:29:25 2013

    [102] getDVPortgroupInfo(4280): failed to get object properties for pg mor. returning

21) Event:E_DEBUG, length:216, at 81693 usecs after Tue Feb 12 00:29:25 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

22) Event:E_DEBUG, length:57, at 81648 usecs after Tue Feb 12 00:29:25 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

23) Event:E_DEBUG, length:81, at 794658 usecs after Tue Feb 12 00:28:53 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

24) Event:E_DEBUG, length:103, at 893290 usecs after Tue Feb 12 00:25:49 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

25) Event:E_DEBUG, length:81, at 872872 usecs after Tue Feb 12 00:25:49 2013

    [102] waitForTask(5983): Failed to wait for updates on the Task. UpdateSet is NULL

26) Event:E_DEBUG, length:271, at 872863 usecs after Tue Feb 12 00:25:49 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

27) Event:E_DEBUG, length:54, at 872795 usecs after Tue Feb 12 00:25:49 2013

    [102] waitForUpdates(5909): ERROR: in WaitForUpdates...

28) Event:E_DEBUG, length:81, at 872468 usecs after Tue Feb 12 00:25:49 2013

    [102] vms_dequeue: (Error) SYSERR_FU_xx: 0x10, err_num (16) in fu_priority_select

29) Event:E_DEBUG, length:81, at 872140 usecs after Tue Feb 12 00:25:49 2013

    [102] vms_dequeue: (Error) SYSERR_FU_xx: 0x10, err_num (16) in fu_priority_select

30) Event:E_DEBUG, length:271, at 869803 usecs after Tue Feb 12 00:24:59 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

31) Event:E_DEBUG, length:84, at 869779 usecs after Tue Feb 12 00:24:59 2013

    [102] getDVPortgroupInfo(4280): failed to get object properties for pg mor. returning

32) Event:E_DEBUG, length:271, at 869764 usecs after Tue Feb 12 00:24:59 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

33) Event:E_DEBUG, length:57, at 869692 usecs after Tue Feb 12 00:24:59 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

34) Event:E_DEBUG, length:81, at 482043 usecs after Tue Feb 12 00:24:11 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

35) Event:E_DEBUG, length:69, at 481930 usecs after Tue Feb 12 00:24:11 2013

    [102] vms_connect(1837): Failed to remove visibility role (0x41b1000d)

36) Event:E_DEBUG, length:106, at 481876 usecs after Tue Feb 12 00:24:11 2013

    [102] vms_vim_delete_visibility_role(3741): Error occurred while searching for visibility r

ole (0x41b1000d)

37) Event:E_DEBUG, length:216, at 481870 usecs after Tue Feb 12 00:24:11 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

38) Event:E_DEBUG, length:216, at 481827 usecs after Tue Feb 12 00:24:11 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

39) Event:E_DEBUG, length:57, at 481785 usecs after Tue Feb 12 00:24:11 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

40) Event:E_DEBUG, length:271, at 867776 usecs after Tue Feb 12 00:23:39 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

41) Event:E_DEBUG, length:60, at 867753 usecs after Tue Feb 12 00:23:39 2013

    [102] waitForTask(6001): ERROR in Retreiving taskInfo from VC

42) Event:E_DEBUG, length:271, at 867746 usecs after Tue Feb 12 00:23:39 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

43) Event:E_DEBUG, length:271, at 867713 usecs after Tue Feb 12 00:23:39 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

44) Event:E_DEBUG, length:57, at 867637 usecs after Tue Feb 12 00:23:39 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

45) Event:E_DEBUG, length:103, at 865970 usecs after Tue Feb 12 00:22:49 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

46) Event:E_DEBUG, length:271, at 865958 usecs after Tue Feb 12 00:22:49 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

47) Event:E_DEBUG, length:71, at 865876 usecs after Tue Feb 12 00:22:49 2013

    [102] updateDVSBySwitchUUID(3945): DVSPerformProductSpecOperation failed

48) Event:E_DEBUG, length:271, at 866456 usecs after Tue Feb 12 00:21:59 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

49) Event:E_DEBUG, length:84, at 866432 usecs after Tue Feb 12 00:21:59 2013

    [102] getDVPortgroupInfo(4280): failed to get object properties for pg mor. returning

50) Event:E_DEBUG, length:271, at 866418 usecs after Tue Feb 12 00:21:59 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

51) Event:E_DEBUG, length:57, at 866345 usecs after Tue Feb 12 00:21:59 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

52) Event:E_DEBUG, length:81, at 694641 usecs after Tue Feb 12 00:20:24 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

53) Event:E_DEBUG, length:81, at 868900 usecs after Tue Feb 12 00:17:24 2013

    [102] vms_dequeue: (Error) SYSERR_FU_xx: 0x10, err_num (16) in fu_priority_select

54) Event:E_DEBUG, length:81, at 865742 usecs after Tue Feb 12 00:16:34 2013

    [102] vms_dequeue: (Error) SYSERR_FU_xx: 0x10, err_num (16) in fu_priority_select

55) Event:E_DEBUG, length:271, at 872408 usecs after Tue Feb 12 00:15:44 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

56) Event:E_DEBUG, length:84, at 872390 usecs after Tue Feb 12 00:15:44 2013

    [102] getDVPortgroupInfo(4280): failed to get object properties for pg mor. returning

57) Event:E_DEBUG, length:271, at 872378 usecs after Tue Feb 12 00:15:44 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

58) Event:E_DEBUG, length:57, at 872324 usecs after Tue Feb 12 00:15:44 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

59) Event:E_DEBUG, length:81, at 701396 usecs after Tue Feb 12 00:14:26 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

60) Event:E_DEBUG, length:69, at 701325 usecs after Tue Feb 12 00:14:26 2013

    [102] vms_connect(1837): Failed to remove visibility role (0x41b1000c)

61) Event:E_DEBUG, length:106, at 701209 usecs after Tue Feb 12 00:14:26 2013

    [102] vms_vim_delete_visibility_role(3741): Error occurred while searching for visibility r

ole (0x41b1000c)

62) Event:E_DEBUG, length:104, at 701205 usecs after Tue Feb 12 00:14:26 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: (null) [VMware vCenter Server 5.0.0

build-623373]

63) Event:E_DEBUG, length:86, at 564869 usecs after Tue Feb 12 00:14:26 2013

    [102] vms_ppm_sync_notify_cback(3897): No active connection to vc to update opaque data

64) Event:E_DEBUG, length:103, at 511811 usecs after Tue Feb 12 00:14:26 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

65) Event:E_DEBUG, length:81, at 471724 usecs after Tue Feb 12 00:14:26 2013

    [102] waitForTask(5983): Failed to wait for updates on the Task. UpdateSet is NULL

66) Event:E_DEBUG, length:216, at 471716 usecs after Tue Feb 12 00:14:26 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

67) Event:E_DEBUG, length:54, at 471679 usecs after Tue Feb 12 00:14:26 2013

    [102] waitForUpdates(5909): ERROR: in WaitForUpdates...

68) Event:E_DEBUG, length:102, at 285651 usecs after Tue Feb 12 00:13:55 2013

    [102] sf_mgmt_srv_updt_port_profile_desc(656): sf_mgmt_srv_lib_updt_dv_port_profile failed.

return fail

69) Event:E_DEBUG, length:226, at 266924 usecs after Tue Feb 12 00:13:55 2013

    [102] populate_err_with_vcinfo(732):  ERROR:  [VMware vCenter Server 5.0.0 build-623373] Ca

nnot set number of ports to 0 because at least 4 out of 4 ports are still in use. A specified p

arameter was not correct.

spec.numPorts.

70) Event:E_DEBUG, length:88, at 266909 usecs after Tue Feb 12 00:13:55 2013

    [102] isTaskComplete(6077):  ERROR: A specified parameter was not correct.

spec.numPorts

71) Event:E_DEBUG, length:104, at 897254 usecs after Tue Feb 12 00:13:04 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: (null) [VMware vCenter Server 5.0.0

build-623373]

72) Event:E_DEBUG, length:271, at 841526 usecs after Tue Feb 12 00:13:04 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

73) Event:E_DEBUG, length:271, at 841496 usecs after Tue Feb 12 00:13:04 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

74) Event:E_DEBUG, length:57, at 841424 usecs after Tue Feb 12 00:13:04 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

75) Event:E_DEBUG, length:70, at 751467 usecs after Tue Feb 12 00:12:15 2013

    [102] vms_vim_rename_dvs_folder(3144): RenameDVSFolder operation failed

76) Event:E_DEBUG, length:39, at 751445 usecs after Tue Feb 12 00:12:15 2013

    [102] Rename(2849): Rename Task failed

77) Event:E_DEBUG, length:103, at 651647 usecs after Tue Feb 12 00:11:43 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

78) Event:E_DEBUG, length:83, at 651640 usecs after Tue Feb 12 00:11:43 2013

    [102] updateDVSBySwitchUUID(3907): failed to retrieve dvs info by uuid. returning -1

79) Event:E_DEBUG, length:216, at 651634 usecs after Tue Feb 12 00:11:43 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

80) Event:E_DEBUG, length:77, at 651612 usecs after Tue Feb 12 00:11:43 2013

    [102] getDVSInfoBySwitchUUID(4200): failed to get object properties. returning

81) Event:E_DEBUG, length:216, at 651598 usecs after Tue Feb 12 00:11:43 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

82) Event:E_DEBUG, length:57, at 651556 usecs after Tue Feb 12 00:11:43 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

83) Event:E_DEBUG, length:81, at 277011 usecs after Tue Feb 12 00:11:12 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

84) Event:E_DEBUG, length:69, at 276931 usecs after Tue Feb 12 00:11:12 2013

    [102] vms_connect(1837): Failed to remove visibility role (0x41b1000c)

85) Event:E_DEBUG, length:106, at 276817 usecs after Tue Feb 12 00:11:12 2013

    [102] vms_vim_delete_visibility_role(3741): Error occurred while searching for visibility r

ole (0x41b1000c)

86) Event:E_DEBUG, length:104, at 276813 usecs after Tue Feb 12 00:11:12 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: (null) [VMware vCenter Server 5.0.0

build-623373]

87) Event:E_DEBUG, length:70, at 232084 usecs after Tue Feb 12 00:07:46 2013

    [102] performDVSProductSpecOperation(5490): Couldn't retrieve bundleId

88) Event:E_DEBUG, length:65, at 232079 usecs after Tue Feb 12 00:07:46 2013

    [102] getDVSBundleId(5322): coudn't retrieve bundleID. return null

89) Event:E_DEBUG, length:216, at 232068 usecs after Tue Feb 12 00:07:46 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

90) Event:E_DEBUG, length:57, at 232014 usecs after Tue Feb 12 00:07:46 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

91) Event:E_DEBUG, length:103, at 521862 usecs after Tue Feb 12 00:07:14 2013

    [102] sf_mgmt_srv_lib_update_dvswitch(1434): updating dvs by switch uuid failed. Sending sy

slog message.

92) Event:E_DEBUG, length:83, at 521848 usecs after Tue Feb 12 00:07:14 2013

    [102] updateDVSBySwitchUUID(3907): failed to retrieve dvs info by uuid. returning -1

93) Event:E_DEBUG, length:216, at 521829 usecs after Tue Feb 12 00:07:14 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

94) Event:E_DEBUG, length:77, at 521807 usecs after Tue Feb 12 00:07:14 2013

    [102] getDVSInfoBySwitchUUID(4200): failed to get object properties. returning

95) Event:E_DEBUG, length:216, at 521793 usecs after Tue Feb 12 00:07:14 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

96) Event:E_DEBUG, length:57, at 521752 usecs after Tue Feb 12 00:07:14 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

97) Event:E_DEBUG, length:216, at 401892 usecs after Tue Feb 12 00:06:43 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

98) Event:E_DEBUG, length:84, at 401869 usecs after Tue Feb 12 00:06:43 2013

    [102] getDVPortgroupInfo(4280): failed to get object properties for pg mor. returning

99) Event:E_DEBUG, length:216, at 401855 usecs after Tue Feb 12 00:06:43 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

100) Event:E_DEBUG, length:57, at 401812 usecs after Tue Feb 12 00:06:43 2013

    [102] getObjectProperties(2344): RetrieveProperties failed

101) Event:E_DEBUG, length:81, at 56168 usecs after Tue Feb 12 00:06:12 2013

    [102] sf_mgmt_srv_get_all_dvpgs(393): dvs = null or no dvpgs exist. returning fail

102) Event:E_DEBUG, length:69, at 55988 usecs after Tue Feb 12 00:06:12 2013

    [102] vms_connect(1837): Failed to remove visibility role (0x41b1000c)

103) Event:E_DEBUG, length:106, at 55827 usecs after Tue Feb 12 00:06:12 2013

    [102] vms_vim_delete_visibility_role(3741): Error occurred while searching for visibility r

ole (0x41b1000c)

104) Event:E_DEBUG, length:104, at 55821 usecs after Tue Feb 12 00:06:12 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: (null) [VMware vCenter Server 5.0.0

build-623373]

105) Event:E_DEBUG, length:83, at 962201 usecs after Tue Feb 12 00:06:11 2013

    [102] vms_vim_create_host_filter(2456): Failed to create a filter with the host spec

106) Event:E_DEBUG, length:216, at 962195 usecs after Tue Feb 12 00:06:11 2013

    [102] convert_soap_fault_to_err(1858): SOAP 1.1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.Timeout. SSL

connect failed in tcp_connect()

107) Event:E_DEBUG, length:49, at 962148 usecs after Tue Feb 12 00:06:11 2013

    [102] createFilter(5838): Error in creating filter

108) Event:E_DEBUG, length:81, at 801167 usecs after Tue Feb 12 00:02:09 2013

    [102] vms_dequeue: (Error) SYSERR_FU_xx: 0x10, err_num (16) in fu_priority_select

109) Event:E_DEBUG, length:271, at 799785 usecs after Tue Feb 12 00:02:09 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

110) Event:E_DEBUG, length:60, at 799762 usecs after Tue Feb 12 00:02:09 2013

    [102] waitForTask(6001): ERROR in Retreiving taskInfo from VC

111) Event:E_DEBUG, length:271, at 799754 usecs after Tue Feb 12 00:02:09 2013

    [102] convert_soap_fault_to_err(1858): Error -1 fault: SOAP-ENV:Client [VMware vCenter Serv

er 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file o

r no input: Operation interrupted or timed out after 160s send or 160s receive delay.

N1KV-PROD-VSM# sh module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

3    248    Virtual Ethernet Module           NA                  ok

4    248    Virtual Ethernet Module           NA                  ok

5    248    Virtual Ethernet Module           NA                  ok

6    248    Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw     

---  ------------------  ------------------------------------------------ 

1    4.2(1)SV2(1.1)      0.0                                             

2    4.2(1)SV2(1.1)      0.0                                             

3    4.2(1)SV2(1.1)      VMware ESXi 5.0.0 Releasebuild-474610 (3.0)     

4    4.2(1)SV2(1.1)      VMware ESXi 5.0.0 Releasebuild-469512 (3.0)     

5    4.2(1)SV2(1.1)      VMware ESXi 5.0.0 Releasebuild-474610 (3.0)     

6    4.2(1)SV2(1.1)      VMware ESXi 5.0.0 Releasebuild-474610 (3.0)     

N1KV-PROD-VSM# 2013 Feb 12 00:36:34 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VMware vCenterError: [VMware vCenter Server 5.0.0 build-623373] Operation could not be completed due to connection failure.End of file or no input: Operation interrupted or timed out after 160s send or 160s receive delay.

2013 Feb 12 00:36:34 N1KV-PROD-VSM %VMS-3-CONNECTION_ERROR: Unable to communicate with vCenter Server/ESX. Disconnecting..

2013 Feb 12 00:36:34 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected from the vCenter Server.

2013 Feb 12 00:36:35 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'MGMT_Uplink' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:36:35 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'Unused_Or_Quarantine_Uplink' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:36:35 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'VM_Data_Uplink' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:36:35 N1KV-PROD-VSM %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'V_L3_CONTROL_99' on the vCenter Server failed. Please  verify port-profile config.

2013 Feb 12 00:36:35 N1KV-PROD-VSM %VMS-5-CONN_CONNECT: Connection 'PROD-VC' connected to the vCenter Server.

2013 Feb 12 00:37:37 N1KV-PROD-VSM %VMS-3-DVS_UPDATE_ERROR: Unable to Update N1KV DVS in the VMware vCenterError: Error in creating Task filter

N1KV-PROD-VSM#

N1KV-PROD-VSM# 2013 Feb 12 00:38:09 N1KV-PROD-VSM %VMS-5-CONN_DISCONNECT: Connection 'PROD-VC' disconnected from the vCenter Server.

N1KV-PROD-VSM# sh svs connections

connection PROD-VC:

    ip address: 192.168.10.10

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: CIT

    admin:  

    max-ports: 8192

    DVS uuid: 57 b5 01 50 b3 f6 66 b4-e1 be a3 2a 97 4a f2 11

    config status: Enabled

    operational status: Disconnected

    sync status: -

    version: -

    vc-uuid: -

N1KV-PROD-VSM# sh svs domain

SVS domain config:

  Domain id:    30 

  Control vlan:  NA

  Packet vlan:   NA

  L2/L3 Control mode: L3

  L3 control interface: mgmt0

  Status: Config push to VC successful.

  Control type multicast: No

Note: Control VLAN and Packet VLAN are not used in L3 mode

N1KV-PROD-VSM#

4 Replies 4

ray.deasy
Level 1
Level 1

Hi,

I got this one sorted, Turned out to be a sub-optimal routing issue between the N1kv MGMT & vCenter affecting the SVS connection. I shutdown one of our  L2/L3 links out from our UCS/N5K systems to an area where vCenter was in production and it stabilised. Just need to fix it now and re-activate the link.

No connection drop off's since this.

Hopefully this may assist someone who sees this type of behaviour in the future,

Regards,

Ray.

Please provide additional details about the underlying issue and the fix you were able to implement. Thanks.

Hello Raymond,

Can you please help me understand what you mean by "I shutdown one of our  L2/L3 links out from our UCS/N5K systems to an area where vCenter was in production and it stabilised. Just need to fix it now and re-activate the link."

Thank you.

With regards,

Nithin.

admin11111
Level 4
Level 4


I am seeijnt this on VSphere 5.5 and I have no idea how to resolve or troubleshoot.

Any help would be most welcome. Below are the log messages.

v1000 errors.PNG

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: