cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1493
Views
0
Helpful
3
Replies

PI 3.2 process takes all resources

m.hegeraat
Level 1
Level 1

Customer find his PI is extremely slow, in the GUI and in the CLI.

 

Exact PI version is 3.2.1

 

Investigation using top, shows a process is taking 160% CPU and 40% memory.

Load average is 5.20, 6.20 4.00

 

On other PI systems this process also takes a very large amount of memory (+-40%), but the CPU usage is much lower 5-10%.

 

Server meets memory CPU and disk requirements.

Checking the IOPS shows the machine exceeds the minimal  200MB/s

 

vm-cpi/admin# ncs run test iops

Testing disk write speed ...

8388608+0 records in

8388608+0 records out

8589934592 bytes (8.6 GB) copied, 32.1145 s, 267 MB/s

 

Checking the pid  using ps -ef | grep <pid> shows the process causing the issue:

 

prime     <pid>     1 99 16:15 ?        00:39:45 /opt/CSCOlumos/bin/pi-java -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=256m -Dinstall.dir=/opt/CSCOlumos -Dapplication.name=ncs-server -Daes.dir=/opt/CSCOlumos -Dmaps.dir=/opt/CSCOlumos/domainmaps -Dheatmap.dir=/opt/CSCOlumos/domainmaps/heatmaps -Dmibs.root=/opt/CSCOlumos/resources/wireless/mibs -Daxis.ClientConfigFile=/opt/CSCOlumos/conf/rfm/classes/com/cisco/common/ha/config/client-config.wsdd -Daxis.socketSecureFactory=org.apache.axis.components.net.SunFakeTrustSocketFactory -Dreport.server=false -Dcom.cisco.xmp.inventory.collector.logging.level=info -Dcom.cisco.xmp.xde.logging.level=info -Dcom.cisco.xmp.nice.logging.level=info -Dcom.cisco.xmp.topology.logging.level=info -Dcom.cisco.xmp.job.manager.logging.level=info -Dcom.cisco.xmp.preferences.logging.level=info -Dcom.cisco.nm.discovery.logging.level=info -Dcom.cisco.xmp.usermgmt.logging.level=info -Dcom.cisco.xmp.grouping.logging.level=info -Dcom.cisco.xmp.persistence.logging.level=info -Dcom.cisco.xmp.config.logging.level=info -Dcom.cisco.xmp.poller.logging.level=info -Dcom.cisco.xmp.audit.logging.level=info -Dcom.cisco.xmp.grt.logging.level=info -Dcom.cisco.xmp.persistence.dbmigration.logging.level=info -Dcom.cisco.xmp.datacenter.eventcorrelation.logging.level=info -Dcom.cisco.xmp.existence.inventory.logging.level=info -Dxmp.default.home=/opt/CSCOlumos/ -Dxmp.updates.home=/opt/CSCOlumos/updates -Dxmp.updates.outofboxtechpacks.home=/opt/CSCOlumos/updates/staging/outofboxtechpacks -Dxmp.update.config.file=/opt/CSCOlumos/conf/update.properties -Dxmp.update.log.dir=/opt/CSCOlumos/logs -Dxmp.update.mdfid=/opt/CSCOlumos/conf/update.mdfid.properties -Dxmp.java.process.name=StartXMP -Djava.system.class.loader=com.cisco.xmp.classLoader.XMPSystemClassLoader -Dcom.cisco.xmp.classLoader.XMPSystemClassLoader.verbose=true -Dcom.cisco.xmp.acdr.devpackloader.DevicePackageLoader.packagesDir=/opt/CSCOlumos/xmp_inventory/dar -Dcom.cisco.xmp.dpl.rest.xar.XarRepository.searchIndexDir=/opt/CSCOlumos/xar-search-index -Dcom.cisco.xmp.inventory.profile.directory=/opt/CSCOlumos/xmp_inventory/dar -Dcom.cisco.nm.workflow.XDEHome=/opt/CSCOlumos/xmp_inventory/xde-home -Dcom.cisco.xmp.xdeinit.loadXARFilesInParallel=false -Dxmp.inventory.config.file=/opt/CSCOlumos/conf/inventory.properties -Dxmp.existenceInventory.config.file=/opt/CSCOlumos/conf/existenceInventory.properties -Dxmp.topology.config.file=/opt/CSCOlumos/conf/topology.properties -Dxmp.persistence.config.file=/opt/CSCOlumos/conf/persistence_config.properties -Dxmp.persistence.optimizeddlmetadata.dir=/opt/CSCOlumos/conf/ddlmetadata -Dxmp.persistence.prunemetadata.dir=/opt/CSCOlumos/conf/prunemetadata -Dxmp.persistence.schemacreate.listeners.dir=/opt/CSCOlumos/conf/schemacreate_listeners -Dxmp.persistence.notificationmetadata.dir=/opt/CSCOlumos/conf/notificationmetadata -Dxmp.db.config.file=/opt/CSCOlumos/conf/jdbc.properties -Dderby.system.home=/opt/CSCOlumos/conf/xmp_data/derbydata -Djava.util.prefs.PreferencesFactory=com.cisco.xmp.prefs.db.impl.DBImplPreferencesFactory -Dxmp.persistence.tqexclusion.file=/opt/CSCOlumos/conf/TqNotRequired.txt -Ddecap.home=/opt/CSCOlumos/decap -Dcom.cisco.xmp.XMPHome=/opt/CSCOlumos -Dcom.cisco.xmp.PI_TYPE=primary -Dxmp.logging.dir=/opt/CSCOlumos/logs -Dxmp.mbc.config.file=/opt/CSCOlumos/conf/mbc_config.properties -Dxmp.mbc.globalobject.types.file=/opt/CSCOlumos/conf/mbc_globalobject_types.properties -Dxmp.mbc.customtags.file=/opt/CSCOlumos/conf/mbc_customtags.properties -Dcom.cisco.xmp.ModelHome=/opt/CSCOlumos/lib/model -Dxmp.conf.dir=/opt/CSCOlumos/conf -Dcom.tc.productkey.path=/opt/CSCOlumos/conf/license/terracotta-license.key -Dtp_persistent_path=/opt/CSCOlumos/licenses -Dxmp.nbi.notifications.config.file=/opt/CSCOlumos/conf/nbi_notifications.properties -Dxmp.nbi.notifications.requiredClasses.file=/opt/CSCOlumos/conf/nbi_notifications_required.txt -Dxmp.poller.config.dir=/opt/CSCOlumos/conf/pollingconfig -Dxmp.grouping.config.file=/opt/CSCOlumos/conf/grouping.properties -Dxmp.grt.config.file=/opt/CSCOlumos/conf/grt_config.properties -Dxmp.messaging.config.file=/opt/CSCOlumos/conf/messaging.properties -Dxmp.update.asd=/opt/CSCOlumos/conf/update.asd.properties -Dxmp.syslog.config.file=/opt/CSCOlumos/conf/syslog_config.properties -Dxmp.updates.home=/opt/CSCOlumos/updates -Dxmp.update.config.file=/opt/CSCOlumos/conf/update.properties -Dxmp.update.mdfid=/opt/CSCOlumos/conf/update.mdfid.properties -javaagent:/opt/CSCOlumos/lib/xmp-third-party/aspectjweaver-1.8.4.jar -Daj.weaving.loadersToSkip=com.cisco.nm.xms.xdi.util.prmgr.ZipLoader,org.apache.jasper.servlet.JasperLoader,net.sf.jasperreports.engine.util.JRClassLoader -Djava.security.egd=file:///dev/urandom -Djavax.net.ssl.trustStore=/opt/CSCOlumos/conf/truststore -XX:CompileCommand=quiet -XX:CompileCommand=exclude,org/snmp4j/Snmp.send -XX:-ReduceInitialCardMarks -Dnet.sf.ehcache.enableShutdownHook=true -Dxmp.jobapprover.config.file=/opt/CSCOlumos/conf/jobapprover.properties -Dxmp.credential_mgmt.config.file=/opt/CSCOlumos/conf/credential_notifications.properties -Dxmp.topology.device.config.file=/opt/CSCOlumos/conf/DeviceImagesMapFile.xml -Dorg.apache.cxf.io.CachedOutputStream.OutputDirectory=/opt/CSCOlumos/tmp -Dxmp.audit.classes.file=/opt/CSCOlumos/conf/ClassesToAudit.xml -Dxmp.usermgmt.config.file=/opt/CSCOlumos/conf/usermgmt.properties -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.reflect.inflationThreshold=2147483647 -Doracle.jdbc.autoCommitSpecCompliant=false -Djdk.tls.ephemeralDHKeySize=2048 -Dxmp.scheduler.config.file=/opt/CSCOlumos/conf/scheduler.properties -Xmx4096m -XX:MetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/CSCOlumos/crash -XX:ThreadStackSize=512 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=5 -XX:NewRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:-UseLoopPredicate -DsolrHome=/opt/CSCOlumos/conf/ifm/ciscoSolrHome2 -DsolrCore=core0 -Difm_config_dir=/opt/CSCOlumos/conf/ifm/ -DLogConfigurationArchiveStackTraces=false -DLogConfigurationArchiveDetailedDebugMsgs=false -DperfPoller.batchSize=1000 -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:MetaspaceSize=1280m -Xms2048m -Xmx6656m -Djava.io.tmpdir=/opt/CSCOlumos/tmp/temp/reports -Duser.timezone=CET -Dnet.sf.ehcache.skipUpdateCheck=true -cp /opt/CSCOlumos/classloader-conf:/opt/CSCOlumos/lib/xmp/XMPClassLoader-11.0.1.jar -Djava.system.class.loader=com.cisco.xmp.classLoader.XMPParallelSystemClassLoader com.cisco.xmp.startup.StartXMP -platformConfFile=xmp-server.properties

 

Possibly this process is called StartXMP 

Does anyone have a clue what this process does?

Why it can suddely take so much CPU ?

 

Thanks for aany feedback or suggestion

 

Michel

1 Accepted Solution

Accepted Solutions

Restarting the systems did not resolve the slowness, but it appears that now after a restore of a backup prime is responding 'normal' again.

This perhaps means that the CPU usage (160%) of the mentioned process,.... is 'normal'.

 

Unfortunately I do not have access to sufficient other PI systems to confirm this.

View solution in original post

3 Replies 3

marce1000
VIP
VIP

 

>Server meets memory CPU and disk requirements

- How is this determined and or is this a VM or an appliance ? ; if it's a VM check cpu and memory resources through vcenter statistics (e.g.)



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

It's a VM. The validation is just from the server definition in Vsphere which matches with:  https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-2/quickstart/guide/cpi_qsg.html#pgfId-121806

 

16 vCPU, 16 GB and 900GB disk.

 

It appears there are no jobs running (I assume they should have an In_Progress state in the job dashboard when they are running)

 

I'm trying to understand what this process is and what it does.

 

And of course I'm looking what could have changed.

Restarting the systems did not resolve the slowness, but it appears that now after a restore of a backup prime is responding 'normal' again.

This perhaps means that the CPU usage (160%) of the mentioned process,.... is 'normal'.

 

Unfortunately I do not have access to sufficient other PI systems to confirm this.