High CPU util with CSR1000v 16.7.1 on KVM

MartinCechvala · ‎02-26-2018

Hello guys,

I'm testing new csr1000v 16.7.1 on KVM under Ubuntu. I was using csr1000v-universalk9.03.12.00.S.154-2.S before without any issues on the same host.

With 16.7.1 version I see 100% CPU utilization all the time (I assigned one vCPU).

show processes cpu plat sort shows 4% util for 5sec/1min/5min, all taken by ucode_pkt_PQF0. The router is doing nothing, I just booted a newly installed csr1000v without any config. I tried both qcow2 image and also tried .iso install.

I'm running KVM on 2x e5-2620v4 xeon with 64GB ram and raid 1 ssd. My libvirt config is very basic (attached).

Edit: this is how I run the VM: qemu-system-x86_64 -enable-kvm -name r3_test -S -machine pc-i440fx-xenial,accel=kvm,usb=off -cpu Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 4096 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a0bc34d2-26fa-458c-b542-fc850e55081d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-r3_test/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/aaaa/r3_test.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/home/aaaa/csr1000v-universalk9.16.07.01.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5d:4f:91,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5907,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -vnc 127.0.0.1:8 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

Is there something I can do to lower the utilization? Any advice is high appreciated.

thanks.

Bela Mareczky · ‎04-25-2018

Dear Martin! I have just faced with this problem also using Cisco CSR1000v 16.06.03 on KVM. Do You have any updates about this issue? The situation is almost the same as Yours: recently booted router with no configuration. The output of the "show proc cpu platform sorted" command shows that, process "ucode_pkt_PQF0" seems to be eating the CPU, but I cannot figure out what could be the reason. Router#show proc cpu platform sorted CPU utilization for five seconds: 18%, one minute: 31%, five minutes: 36% Core 0: CPU utilization for five seconds: 18%, one minute: 30%, five minutes: 34% Pid PPid 5Sec 1Min 5Min Status Size Name -------------------------------------------------------------------------------- 27451 26449 31% 31% 31% S 1107722240 ucode_pkt_PQF0 2311 24507 1% 1% 2% S 4894691328 linux_iosd-imag 32424 30847 0% 0% 0% S 4665344 sleep Thank U very much in advance! Best Regards, Belabacsi

sp2720401 · ‎04-27-2018

I have the same problem with any of the Denali, Everest, or Fuji releases. I have a router without a startup configuration and all interfaces admin down. The VM eats about 50 percent of the CPU no matter how many cores I throw at it.

MartinCechvala · ‎04-28-2018

hello Bela,

I tried policing the cpu using cgroupson the host but that caused the XE to throw errors/kernel panics/etc - not stable. I also tried pinning 10 XEs on one core, to naturally limit the cpu util of each XE. This works, the console response of XE is a bit slower but it allowed me to run 100 XEs easily (I didn't need more). Also a solution is to use previos release, 15.x.x if you don't need newest features - this is what I'm doing now. Use older release when I can and new release with pinning 5-10 XEs on the same core when I need to use the new release.

These are not solutions, just workarounds though...

Hope it helps :)