UCS C200 M2 ESXi Connectivity Issues

[abattard1]

We have a UCS C200 M2 running ESXi 4.  I have Cisco Unity Connection 8.5 and two Call Manager 8.5 subscribers running on it.  Unity Connection is running fine, however the two subscribers are running extremely slow when web browsing to them.  All vm's were created using the ova template from the cisco dokwiki.  CPU and disk utilization appears to be normal for those vm's.  A bigger problem is that we are no longer able to connect to our esxi management network.  After rebooting the host server we were able to reconnect to it, but the problem has since come back again.  A vmware expert had told me that the linux db went into a read-only state due to hardware storage issues.  He said he had seen this on another Cisco UCS server.  Has anyone else encountered this issue or know where the problem may lie.  I believe that we may have two seperate issues here.

[abattard1]

I was able to find the cause of this problem.  There is a known vmware bug in esxi version 4.1.  The bug will cause the MAINSYS to run at 100% utilization, which in turn causes the ESX DB to enter a read only state.  Below is the link to the KB article and also a log of the before and after the fix.  They found this problem a couple weeks ago, so there may already be a hotfix for it.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2001457

/var/log/ipmi/0 # vdf -h

Tardisk                 Space      Used

SYS1                     201M      201M

SYS2                      55M       55M

SYS3                       1M        1M

SYS4                      12K       12K

SYS5                      12K       12K

SYS6                      16K       16K

-----

Ramdisk                  Size      Used Available Use% Mounted on

MAINSYS                   32M      32M        0B 100% --

tmp                      192M        0B     192M   0% --

updatestg                750M        0B     750M   0% --

hostdstats               107M        2M     104M   2% --

------------FIX-------------------------

/var/log/ipmi/0 # rm sel.raw

/var/log/ipmi/0 # rm sel

/var/log/ipmi/0 # vdf -h

Tardisk                 Space      Used

SYS1                     201M      201M

SYS2                      55M       55M

SYS3                       1M        1M

SYS4                      12K       12K

SYS5                      12K       12K

SYS6                      16K       16K

-----

Ramdisk                  Size      Used Available Use% Mounted on

MAINSYS                   32M       1M       30M   4% --

tmp                      192M        0B     192M   0% --

updatestg                750M        0B     750M   0% --

hostdstats               107M       2M      104M   2% --

[dakeller]

This solution looks reasonable. But keep an eye on the system to see if the MAINSYS starts to creep back up.  There are some other possible causes that are not exclusively identified here. 

I can tell you for sure that if the CPU on the host is maxed out, CUCM will go into read only. 

Thanks,

Dan Keller

Technical Marketing Engineer

[cgoldsmith]

Does anyone know if there is a long term patch available to resolve this?  I don't like vmware's solution, turning off IPMI support is not a good thing if we want to monitor our hardware.

Thanks

[dakeller]

Charles,

This question can be answered by Cisco. The stated issue and resolution is coming from VMWare.  Cisco would not be able to comment on a long term solution for the issue because Cisco does not own the code for ESXi host.  I would request the long term solution guidance from VMWare.

Thanks,

Dan Keller

Technical Marketing Engineer