Customer find his PI is extremely slow, in the GUI and in the CLI.
Exact PI version is 3.2.1
Investigation using top, shows a process is taking 160% CPU and 40% memory.
Load average is 5.20, 6.20 4.00
On other PI systems this process also takes a very large amount of memory (+-40%), but the CPU usage is much lower 5-10%.
Server meets memory CPU and disk requirements.
Checking the IOPS shows the machine exceeds the minimal 200MB/s
vm-cpi/admin# ncs run test iops
Testing disk write speed ...
8388608+0 records in
8388608+0 records out
8589934592 bytes (8.6 GB) copied, 32.1145 s, 267 MB/s
Checking the pid using ps -ef | grep <pid> shows the process causing the issue:
prime <pid> 1 99 16:15 ? 00:39:45 /opt/CSCOlumos/bin/pi-java -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=256m -Dinstall.dir=/opt/CSCOlumos -Dapplication.name=ncs-server -Daes.dir=/opt/CSCOlumos -Dmaps.dir=/opt/CSCOlumos/domainmaps -Dheatmap.dir=/opt/CSCOlumos/domainmaps/heatmaps -Dmibs.root=/opt/CSCOlumos/resources/wireless/mibs -Daxis.ClientConfigFile=/opt/CSCOlumos/conf/rfm/classes/com/cisco/common/ha/config/client-config.wsdd -Daxis.socketSecureFactory=org.apache.axis.components.net.SunFakeTrustSocketFactory -Dreport.server=false -Dcom.cisco.xmp.inventory.collector.logging.level=info -Dcom.cisco.xmp.xde.logging.level=info -Dcom.cisco.xmp.nice.logging.level=info -Dcom.cisco.xmp.topology.logging.level=info -Dcom.cisco.xmp.job.manager.logging.level=info -Dcom.cisco.xmp.preferences.logging.level=info -Dcom.cisco.nm.discovery.logging.level=info -Dcom.cisco.xmp.usermgmt.logging.level=info -Dcom.cisco.xmp.grouping.logging.level=info -Dcom.cisco.xmp.persistence.logging.level=info -Dcom.cisco.xmp.config.logging.level=info -Dcom.cisco.xmp.poller.logging.level=info -Dcom.cisco.xmp.audit.logging.level=info -Dcom.cisco.xmp.grt.logging.level=info -Dcom.cisco.xmp.persistence.dbmigration.logging.level=info -Dcom.cisco.xmp.datacenter.eventcorrelation.logging.level=info -Dcom.cisco.xmp.existence.inventory.logging.level=info -Dxmp.default.home=/opt/CSCOlumos/ -Dxmp.updates.home=/opt/CSCOlumos/updates -Dxmp.updates.outofboxtechpacks.home=/opt/CSCOlumos/updates/staging/outofboxtechpacks -Dxmp.update.config.file=/opt/CSCOlumos/conf/update.properties -Dxmp.update.log.dir=/opt/CSCOlumos/logs -Dxmp.update.mdfid=/opt/CSCOlumos/conf/update.mdfid.properties -Dxmp.java.process.name=StartXMP -Djava.system.class.loader=com.cisco.xmp.classLoader.XMPSystemClassLoader -Dcom.cisco.xmp.classLoader.XMPSystemClassLoader.verbose=true -Dcom.cisco.xmp.acdr.devpackloader.DevicePackageLoader.packagesDir=/opt/CSCOlumos/xmp_inventory/dar -Dcom.cisco.xmp.dpl.rest.xar.XarRepository.searchIndexDir=/opt/CSCOlumos/xar-search-index -Dcom.cisco.xmp.inventory.profile.directory=/opt/CSCOlumos/xmp_inventory/dar -Dcom.cisco.nm.workflow.XDEHome=/opt/CSCOlumos/xmp_inventory/xde-home -Dcom.cisco.xmp.xdeinit.loadXARFilesInParallel=false -Dxmp.inventory.config.file=/opt/CSCOlumos/conf/inventory.properties -Dxmp.existenceInventory.config.file=/opt/CSCOlumos/conf/existenceInventory.properties -Dxmp.topology.config.file=/opt/CSCOlumos/conf/topology.properties -Dxmp.persistence.config.file=/opt/CSCOlumos/conf/persistence_config.properties -Dxmp.persistence.optimizeddlmetadata.dir=/opt/CSCOlumos/conf/ddlmetadata -Dxmp.persistence.prunemetadata.dir=/opt/CSCOlumos/conf/prunemetadata -Dxmp.persistence.schemacreate.listeners.dir=/opt/CSCOlumos/conf/schemacreate_listeners -Dxmp.persistence.notificationmetadata.dir=/opt/CSCOlumos/conf/notificationmetadata -Dxmp.db.config.file=/opt/CSCOlumos/conf/jdbc.properties -Dderby.system.home=/opt/CSCOlumos/conf/xmp_data/derbydata -Djava.util.prefs.PreferencesFactory=com.cisco.xmp.prefs.db.impl.DBImplPreferencesFactory -Dxmp.persistence.tqexclusion.file=/opt/CSCOlumos/conf/TqNotRequired.txt -Ddecap.home=/opt/CSCOlumos/decap -Dcom.cisco.xmp.XMPHome=/opt/CSCOlumos -Dcom.cisco.xmp.PI_TYPE=primary -Dxmp.logging.dir=/opt/CSCOlumos/logs -Dxmp.mbc.config.file=/opt/CSCOlumos/conf/mbc_config.properties -Dxmp.mbc.globalobject.types.file=/opt/CSCOlumos/conf/mbc_globalobject_types.properties -Dxmp.mbc.customtags.file=/opt/CSCOlumos/conf/mbc_customtags.properties -Dcom.cisco.xmp.ModelHome=/opt/CSCOlumos/lib/model -Dxmp.conf.dir=/opt/CSCOlumos/conf -Dcom.tc.productkey.path=/opt/CSCOlumos/conf/license/terracotta-license.key -Dtp_persistent_path=/opt/CSCOlumos/licenses -Dxmp.nbi.notifications.config.file=/opt/CSCOlumos/conf/nbi_notifications.properties -Dxmp.nbi.notifications.requiredClasses.file=/opt/CSCOlumos/conf/nbi_notifications_required.txt -Dxmp.poller.config.dir=/opt/CSCOlumos/conf/pollingconfig -Dxmp.grouping.config.file=/opt/CSCOlumos/conf/grouping.properties -Dxmp.grt.config.file=/opt/CSCOlumos/conf/grt_config.properties -Dxmp.messaging.config.file=/opt/CSCOlumos/conf/messaging.properties -Dxmp.update.asd=/opt/CSCOlumos/conf/update.asd.properties -Dxmp.syslog.config.file=/opt/CSCOlumos/conf/syslog_config.properties -Dxmp.updates.home=/opt/CSCOlumos/updates -Dxmp.update.config.file=/opt/CSCOlumos/conf/update.properties -Dxmp.update.mdfid=/opt/CSCOlumos/conf/update.mdfid.properties -javaagent:/opt/CSCOlumos/lib/xmp-third-party/aspectjweaver-1.8.4.jar -Daj.weaving.loadersToSkip=com.cisco.nm.xms.xdi.util.prmgr.ZipLoader,org.apache.jasper.servlet.JasperLoader,net.sf.jasperreports.engine.util.JRClassLoader -Djava.security.egd=file:///dev/urandom -Djavax.net.ssl.trustStore=/opt/CSCOlumos/conf/truststore -XX:CompileCommand=quiet -XX:CompileCommand=exclude,org/snmp4j/Snmp.send -XX:-ReduceInitialCardMarks -Dnet.sf.ehcache.enableShutdownHook=true -Dxmp.jobapprover.config.file=/opt/CSCOlumos/conf/jobapprover.properties -Dxmp.credential_mgmt.config.file=/opt/CSCOlumos/conf/credential_notifications.properties -Dxmp.topology.device.config.file=/opt/CSCOlumos/conf/DeviceImagesMapFile.xml -Dorg.apache.cxf.io.CachedOutputStream.OutputDirectory=/opt/CSCOlumos/tmp -Dxmp.audit.classes.file=/opt/CSCOlumos/conf/ClassesToAudit.xml -Dxmp.usermgmt.config.file=/opt/CSCOlumos/conf/usermgmt.properties -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.reflect.inflationThreshold=2147483647 -Doracle.jdbc.autoCommitSpecCompliant=false -Djdk.tls.ephemeralDHKeySize=2048 -Dxmp.scheduler.config.file=/opt/CSCOlumos/conf/scheduler.properties -Xmx4096m -XX:MetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/CSCOlumos/crash -XX:ThreadStackSize=512 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=5 -XX:NewRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:-UseLoopPredicate -DsolrHome=/opt/CSCOlumos/conf/ifm/ciscoSolrHome2 -DsolrCore=core0 -Difm_config_dir=/opt/CSCOlumos/conf/ifm/ -DLogConfigurationArchiveStackTraces=false -DLogConfigurationArchiveDetailedDebugMsgs=false -DperfPoller.batchSize=1000 -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:MetaspaceSize=1280m -Xms2048m -Xmx6656m -Djava.io.tmpdir=/opt/CSCOlumos/tmp/temp/reports -Duser.timezone=CET -Dnet.sf.ehcache.skipUpdateCheck=true -cp /opt/CSCOlumos/classloader-conf:/opt/CSCOlumos/lib/xmp/XMPClassLoader-11.0.1.jar -Djava.system.class.loader=com.cisco.xmp.classLoader.XMPParallelSystemClassLoader com.cisco.xmp.startup.StartXMP -platformConfFile=xmp-server.properties
Possibly this process is called StartXMP
Does anyone have a clue what this process does?
Why it can suddely take so much CPU ?
Thanks for aany feedback or suggestion
Michel
