Cisco RV340 | CPU 100%

[knollknows]

I am experiencing severe network degradation on client machines and I am having extreme delays to access the admin console of the router.  From a client side perspective, some sites are appearing to load slowly and I am noticing some lag. My environment consists of ~~30 devices from Xbox, PCs and IoT devices.  I reviewed the Cisco "summary" page an I see 100% CPU utilization.  I configured  syslog to a remote server and enabled "debugging". I can share my findings as required. 

I did see some other posts of other users experiencing similar issues and I also can confirm that a reboot will resolve this issue for roughly ~~8 hrs.  I have done a router reset and reconfigured all options to rule out a stuck config from a router upgrade. I did try out the suggested "peer solutions" to lower the IPS/IDS options to "balanced" however this had no impact. 

Model: Cisco RV 340

Firmware:

Serial Number: OMITTED System Up Time: 0 Days 22 Hours 27 Minutes 11 Seconds Current Time: 2019-Dec-13, 07:49:18 PST CPU/Memory Usage: 100% / 37% PID VID: RV340-K9 V01

Firmware Information   Firmware Version: 1.0.03.16

[normh]

Latest update on the TAC Case...  The support engineer has asked me to let the router run over the weekend with the Security license disabled but still on the latest firmware, 1.0.03.16.  So far, I have not seen the problem.  He said next actions Monday will probably be to downgrade the firmware to the previous version, 1.0.03.15, and re-enable the Security license to see if the problem was introduced in the latest firmware (regression testing anyone?).  I will update again after testing this or whatever he comes up with for next steps.

[normh]

After no callback and no updates to the case from Caleb, the Cisco support engineer either yesterday or today, I decided to go ahead and downgrade my RV340 to firmware version 1.0.3.15 and re-enable the Security license.  So far it has behaved normally with no 100% CPU spikes.  I will update this post if this changes.

[normh]

I have gone back to my Netgate SG3100 pfSense firewall for now.  Waiting for Cisco to come up with a fix.  I am tired of dealing with the quirks of this box.  Hopefully the next version of firmware will allow the use of the Security license I paid for.

[normh]

The latest now is that the router spontaneously rebooted again at 4:00am today and reloaded the 1.0.3.16 firmware!  This is despite having auto upgrade turned off for firmware.  I assume the reason why it did this is because the 1.0.3.16 firmware is loaded in the router (and was the active image even though it was booted with 1.0.3.15??).  If there were a way to delete it, I would, but I can't find a way to do it.  Also, I spoke with Caleb just now and he says they have multiple cases open now about this problem.  He thinks it may be due to some changes made in the licensing functionality in 1.0.3.16.

[michaeldowns1000]

Very disconcerting!

Since paying for the security license, and bugs in those functions causing so many problems, i was considering reverting to 1.0.3.15 firmware as a work-around.

That does not seem like a viable option.

normh, thanks for updating this thread!

[JanPlachy0816]

I can confirm, that downgrade to version 1.0.03.15 helped - but the behavior of router/firewall is still strange. Based on detailed network monitoring, the router that was running on version 1.0.03.16 drastically increased its response time after 8 hours (from <1 ms to 20+ ms within one minute, CPU load jumped from approx. 3% to 100%). The similar problem occurs also on version 1.0.03.16 but after 3,5 days. Periodical restart is not solution just only temporarily workaround. Please let us know if you receive any further information from Cisco's support. Thank you in advance.

[normh]

I assume you meant to say 1.0.3.15 exhibits the problem after 3-5 days, not 10.0.3.16.  My router is at 3 days, 4 hours run-time since last boot and is running 1.0.3.15.  No problems yet.  I'll post back whether my unit starts exhibiting the problem in the next 2 days.

[JanPlachy0816]

You're correct. The version 1.0.03.15 seems to be more stable. I wrote it incorrectly. Last period without the need of restart was almost 4 days, but the load grew-up faster and more than usual so I was unable to log-in to the admin interface and reboot it and I had to manually switch-off the router. I've attached statistics for last 14 days. Periods highlighted means response that exceeds 2 ms. I used version 1.0.03.16 until 01/29. Router uptime without problems was stable usually for 8 - 48 hours. response time of the router decreased, but I was always able to log-in and reboot the device. After downgrade, the router was stable for almost 4 days, but after that the response time grew-up approx. 100x and made the admin interface inaccessible.

[JanPlachy0816]

Some news: due internet provider infrastructure problems firewall was unable to re-verify license validity and last 6 days ran without advanced features (web filtering, IPS, AV, etc.) ... without any problem. Average CPU load about 3%, response about 0,8 ms. It leads to the conclusion, that a problem is really in some advanced feature (memory leak or something else). I'm gonna try to run more times with a registered/unregistered license to verify the behavior. 

[knollknows]

Hello everyone,

I see that this thread has gained momentum in having multiple users reporting the same if not similar issues.  The last update from a Cisco 'agent' was in having me call support and make a formal troubleshooting support ticket.  I have NOT done so.

Like previously mentioned, I am simply rebooting the server on a daily basis for now.  Time permitting I will call support and have this 'documented'. 

I can see that many of you are trying or considering downgrading the Firmware as a workaround. I can see the logic behind this but I would decline that option myself as the bug fixes enabled by the current version resolve security/performance issues.  Again I am assuming that this version of firmware is the 'cause' for the CPU utilization to hit 100%

Hopefully a solution comes forward soon. I'll update this thread if further updates become available for the curious minds.

[normh]

I am now at 5 days, 11 hours since last reboot and I have not seen any problems with the Security license and its features enabled.  It looks like, in my case, downgrading the 1.0.3.15 has mitigated the 100% CPU problem.  Don't get me wrong, I do not think this is a satisfactory solution, but it enables me to continue to use the router with the license until Cisco releases a fix.

[AlKor]

I didnt use RV340 as gateway for all network segments, use only for one VLAN with around 20-30 WiFi clients.
System Up Time > 14 Days. CPU/Memory Usage: 1% / 36%. SW 1.0.03.16, Signature 2.0.0.0008.
Try change GW to other switch and change routes. It this case App/Clinet statistic, filtering and so on will stop work (it is other bug, and Cisco can not fix it 6 month) :)
But router will stable work.

[JanPlachy0816]

Move the routing/filtering capability to lower segments (e.g. core switch) is not a system solution. I've bought the router and additional Security license primarily from the reason of single point of rules definition/VLAN management/VPN server/statistics etc., but as I can see, if I would like to have a stable device, I must move advanced (paid) features to other network segments. Nope. Another 1 day, 20 hours without problem, but after that 100% CPU load occurred again and a reboot was required. The only advice from Cisco's side is to "open a new case", but apparently I have no such permission now :( ... I wish it only SSH access working. In this case, I could enforce a device reboot based on slow response events. But this is another Cisco's "joke" - to have a console port on the device, serial-2-LAN cable included in package, BUT "CLI access disabled for feature use". What are they waiting for? Christmas?

[normh]

I have gone back to my Netgate SG3100 pfSense firewall for now.  Waiting for Cisco to come up with a fix.  I am tired of dealing with the quirks of this box.  Hopefully the next version of firmware will allow the use of the Security license I bought.

[normh]

Guess What!  Cisco closed my TAC Case with NO SOLUTION!  Thanks Cisco:

This baby goes on eBay.  I am totally disgusted with Cisco.  Wasted $100 for this Security license that breaks the router and TAC closes the case when the problem has not been fixed.

Here is the email reply to my request for an update:

Hello <my CCO ID Email>,

Your email attempt to update a Service Request (SR) number or Third Party Number (TPN) had one or more failures.  Details provided below.

Note update successful: NONE

Note update (and any file attachments) failed - SR Closed:  688307775

Note update failed - SR Not Found:  NONE

The following file(s) could not be uploaded successfully.  Failure details provided below.

NONE

You can also reach the Cisco Technical Assistance Center via the additional methods found on our Technical Support Worldwide Contacts page at:

http://www.cisco.com/warp/public/687/Directory/DirTAC.shtml

For other questions, please visit our Technical Support Website:

http://www.cisco.com/cisco/web/support/index.html

Sincerely,

Technical Assistance Center

Cisco Systems, Inc.

My email:

Hello Caleb,

I haven’t heard anything further about this.  Can you give me an update on a fix for this problem?

It doesn’t appear that Cisco has much interest in fixing even serious problems with SOHO products.

Thanks