Cisco RV260 stuck freezes stops responding

[emwu90]

For some time now, I've been experiencing issues with RV260 routers. The problem is that, for unknown reasons, the router freezes/stuck and stops responding to ping and others protocol requests on both the WAN and LAN.

The LED's continue to light up (screen in attachment)

I have the latest firmware installed, which is Firmware Version 1.0.01.10.

Issues related to the ISP and local network do not seem to be the cause, as they are functioning correctly.

The only solution is to restart the device, after which it resumes normal operation.

I have attached the latest router logs (debug mode), which have been sent to the syslog server:

Has anyone else encountered a similar problem with this device? I also have RV340 devices, and they work correctly without any freezing issues.

The issue becomes particularly problematic as I only have remote access to the device through VPN and site-to-site connections. In the event of a freeze / stuck, I lose access to the device and local network.

[pieterh]

the RV260 has an option to automaticaly update the firmware
this service has been terminated by Cisco,
maybe this is the cause of the freezes ? -> switch it off
Heads up - Automatically download updates from Cisco.com going away - Cisco Community

[emwu90]

Thank you for the suggestion. We have several of those devices. We will implement this change and monitor the results.

[marce1000]

 - One thing to try is to save the configuration externally ; factory reset and restore configuration ;  check for improvement.

 M.

[emwu90]

Thank you for the response. The issue is that we manage the devices remotely, and a factory reset might be a bit problematic, because we will lose access. However, we will try this solution as much as possible.

[emwu90]

Unfortunately, none of the above suggestions helped. Today, we had another occurrence. Below the latest collected log from the router:

Wed Feb 14 22:45:51 2024; <13>1 2024-02-14T23:45:51+03:00 mwan3track - - [meta sequenceId="73781"] Interface wan1 (eth0) is offline

[pieterh]

It is possible the router has reached the end of it usable lifetime
but there are still some simple actions you can try

1) try to locate another PSU for this device, maybe the problem is originated in the PSU

2) it looks like the device is placed in a data cabinet ?
   -> the router came packed with "ears"/brackets for mounting in a cabinet
   ==>> find those and mount the router to allow air to flow around it
           as it may get to hot sitting on the Hikvision device,
           and there is also a conference mic/phone placed on top of the router?
           else place the router on an separate shelve

[emwu90]

Thank you for your response.

We will try exchanging the power supply.

Regarding the rack cabinet and the lack of space, as well as the age of the devices, it's not just happening with one device but with several that we own. They are placed in different locations, so I wouldn't necessarily associate these issues with each other.

As for the age, the devices were purchased about 2 years ago, so it would be strange if they started malfunctioning due to their age.

[pieterh]

did you buy directly from Cisco ? then you may be able to claim warranty there
End-of-Sale and End-of-Life Announcement for the Cisco RV 160, RV260, RV345P, RV340W, RV260W, RV260P and RV160W VPN Router

[emwu90]

I checked and unfortunately but the devices are out of warranty.

Tonight there were once again two occurrences of the problem, on two independent devices. I was able to collect more logs because after the WAN connection was losse, the router continued to log to syslog locally:

Location 1:
Mon Feb 19 05:37:21 2024;; <4>1 2024-02-19T03:37:21+00:00 kernel - - [meta sequenceId="92683"] [1619208.108750] WAN: Link is Down
Mon Feb 19 05:37:21 2024;; <30>1 2024-02-19T03:37:21+00:00 charon - - [meta sequenceId="92684"] 08[KNL] disappeared from eth0

Mon Feb 19 05:37:22 2024; <30>1 2024-02-19T03:37:22+00:00  charon - - [meta sequenceId="92685"] 05[IKE] old path is not available anymore, try to find another

Mon Feb 19 05:37:22 2024; <30>1 2024-02-19T03:37:22+00:00 - - [meta sequenceId="92686"] 05[IKE] looking for a route to ...Mon Feb 19 05:37:24 2024; <30>1 2024-02-19T03:37:24+00:00 charon - - [meta sequenceId="92703"] 04[NET] error writing to socket: Invalid argument

Location 2:
Mon Feb 19 02:00:56 2024;; <29>1 2024-02-19T03:00:56+03:00 netifd - - [meta sequenceId="83168"] Interface 'wan1' has link connectivity loss
Mon Feb 19 02:00:56 2024;; <30>1 2024-02-19T03:00:56+03:00 charon - - [meta sequenceId="83169"] 05[KNL] disappeared from eth0
Mon Feb 19 02:00:56 2024;; <30>1 2024-02-19T03:00:56+03:00 charon - - [meta sequenceId="83170"] 12[KNL] interface eth0 deactivated
Mon Feb 19 02:00:56 2024;; <30>1 2024-02-19T03:00:56+03:00 charon - - [meta sequenceId="83178"] 05[KNL] interface eth0 activated
Mon Feb 19 02:00:56 2024;; <30>1 2024-02-19T03:00:56+03:00 charon - - [meta sequenceId="83179"] 06[KNL] received netlink error: Network is unreachable (101)
Mon Feb 19 02:01:00 2024; <30>1 2024-02-19T03:01:00+03:00 charon - - [meta sequenceId="83223"] 04[NET] error writing to socket: Invalid argument

--------------------------------------------------------

it looks like there is a temporary break in the connection and when the connection comes back the router is not able to establish it again

[emwu90]

Another occurrence in a different location. This time also managed to collect interesting logs:

Tue Feb 20 01:24:40 2024; ; <4>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63770"] [451172.465816] WAN: Link is Down

Tue Feb 20 01:24:40 2024; ; <29>1 2024-02-20T02:24:40+03:00 netifd - - [meta sequenceId="63771"] Interface 'wan1' has link connectivity loss

Tue Feb 20 01:24:40 2024; ; <30>1 2024-02-20T02:24:40+03:00 charon - - [meta sequenceId="63772"] 08[KNL] disappeared from eth0

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63773"] [451172.492014] eth0: pfe_eth_shutdown

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63774"] [451172.532048] hif_lib_client_unregister : client: be100424, client_id: 0, txQ_depth: 2048, rxQ_depth: 2048

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63775"] [451172.532069] hif_process_client_req: unregister client_id 0

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63776"] [451172.532077] pfe_hif_client_unregister

Tue Feb 20 01:24:40 2024; ; <30>1 2024-02-20T02:24:40+03:00 charon - - [meta sequenceId="63777"] 14[KNL] interface eth0 deactivated

Tue Feb 20 01:24:40 2024; ; <30>1 2024-02-20T02:24:40+03:00 charon - - [meta sequenceId="63778"] 11[KNL] fe80.... disappeared from eth0

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63779"] [451172.549139] eth0: pfe_eth_open

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63780"] [451172.549685] hif_process_client_req: register client_id 0

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63781"] [451172.549695] pfe_hif_client_register

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63782"] [451172.549708] hif_lib_set_rx_cpu_affinity:Client (0) and HIF Rx processing will use same cpu

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63783"] [451172.549721] eth0: pfe_gemac_init

Tue Feb 20 01:24:40 2024; ; <6>1 2024-02-20T02:24:40+03:00 kernel - - [meta sequenceId="63784"] [451172.550121] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Tue Feb 20 01:24:40 2024; ; <30>1 2024-02-20T02:24:40+03:00 charon - - [meta sequenceId="63785"] 12[KNL] interface eth0 activated

[pieterh]

the messages "eth0: link is not ready"  seem to indicate a hardware incident?
this is a physical interface, not the logical connection of/over the VPN
can you match these date/time with maintenance by your ISP to the internet lines ?

[emwu90]

Okay, thank you for the information. We will verify if there were any Internet problems with the provider during that time.

[emwu90]

@pieterh Unfortunately, based on the information received from the ISP during the interruptions and stuck routers, they did not identify any connectivity issues on their end.

[emwu90]

We connected an additional switch between the IPS device and the RV260 router. Unfortunately, the issue reoccurred, but this time, we obtained additional logs because we didn't have to restart the router, only the switch. This allowed us to capture logs from the moment the connection was restored after restarting the switch.

Tue Mar 5 09:27:59 2024; ; <4>1 2024-03-05T10:27:59+03:00 kernel - - [meta sequenceId="328"] [428898.557814] bcm54210s_read_status 356 Status: Try copper link...

Tue Mar 5 09:27:59 2024; ; <4>1 2024-03-05T10:27:59+03:00 kernel - - [meta sequenceId="329"] [428898.558083] Restoring WAN: speed 0xa->0xa, autong 0x1->0x1, duplex 0x0->0x0, advertising 0x7f->0x7f

Tue Mar 5 09:28:00 2024; ; <13>1 2024-03-05T10:28:00+03:00 VPN-timer - - [meta sequenceId="330"] Timer event for s2s. Primary is not reachable. Also backup is not configured...

Tue Mar 5 09:28:01 2024; ; <4>1 2024-03-05T10:28:01+03:00 kernel - - [meta sequenceId="331"] [428900.577821] bcm54210s_read_status 356 Status: Try copper link...

Tue Mar 5 09:28:01 2024; ; <4>1 2024-03-05T10:28:01+03:00 kernel - - [meta sequenceId="332"] [428900.578089] Restoring WAN: speed 0xa->0xa, autong 0x1->0x1, duplex 0x0->0x0, advertising 0x7f->0x7f

Tue Mar 5 09:28:01 2024; ; <13>1 2024-03-05T10:28:01+03:00 VPN-timer - - [meta sequenceId="333"] Timer event for s2s. Primary is not reachable. Also backup is not configured...

Tue Mar 5 09:28:03 2024; ; <4>1 2024-03-05T10:28:03+03:00 kernel - - [meta sequenceId="334"] [428902.597664] bcm54210s_read_status 356 Status: Try copper link...

Tue Mar 5 09:28:03 2024; ; <4>1 2024-03-05T10:28:03+03:00 kernel - - [meta sequenceId="335"] [428902.597932] Restoring WAN: speed 0xa->0xa, autong 0x1->0x1, duplex 0x0->0x0, advertising 0x7f->0x7f

Tue Mar 5 09:28:05 2024; ; <4>1 2024-03-05T10:28:05+03:00 kernel - - [meta sequenceId="336"] [428904.617675] bcm54210s_read_status 452 Status: Copper link up

Tue Mar 5 09:28:05 2024; ; <4>1 2024-03-05T10:28:05+03:00 kernel - - [meta sequenceId="337"] [428904.617944] Restoring WAN: speed 0xa->0xa, autong 0x1->0x1, duplex 0x0->0x0, advertising 0x7f->0x7f

Tue Mar 5 09:28:05 2024; ; <4>1 2024-03-05T10:28:05+03:00 kernel - - [meta sequenceId="338"] [428904.618554] WAN: COPPER Link is Up - 1Gbps/Full - flow control rx/tx

Tue Mar 5 09:28:05 2024; ; <6>1 2024-03-05T10:28:05+03:00 kernel - - [meta sequenceId="339"] [428904.618600] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Tue Mar 5 09:28:05 2024; ; <30>1 2024-03-05T10:28:05+03:00 charon - - [meta sequenceId="340"] 16[KNL] appeared on eth0

Tue Mar 5 09:28:05 2024; ; <133>1 2024-03-05T10:28:05+03:00 system - - [meta sequenceId="342"] Workaround for WAN port issue is executed.

Tue Mar 5 09:28:05 2024; ; <133>1 2024-03-05T10:28:05+03:00 system - - [meta sequenceId="343"] Fiber not link up, donot toggle wan port in zfinish.

Tue Mar 5 09:28:06 2024; ; <30>1 2024-03-05T10:28:06+03:00 charon - - [meta sequenceId="344"] 05[KNL] appeared on eth0

I think they are very interesting especially this one:

Tue Mar 5 09:28:05 2024; ; <133>1 2024-03-05T10:28:05+03:00 system - - [meta sequenceId="342"] Workaround for WAN port issue is executed.