topic Re: NTP broken on WLC 2405 in Wireless

NTP broken on WLC 2405

dmcgrath.ca — Mon, 05 Jul 2021 17:48:14 GMT

Hi,

I have noticed that a WLC 2504 that I have appears to have problems loading an NTP server. Whenever the system boots up, or is attempted to configure a time server, the console errors out with the following logs:

/usr/bin/ntpd: error while loading shared libraries: libmd5.so.0: cannot open shared object file: No such file or directory

Is there any sort of built in file checker? The system is currently running 8.5.151.0, and even exhibited the behaviour in the previous version 8.5.135.0 as well, so upgrading the image in an attempt to fix the problem has also failed.

My concern is there is some corruption in the underlying image that is booted by the system, and I am curious if it can be repaired with any type of re-flashing?

System Information
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.151.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. PIC 20.0

Re: NTP broken on WLC 2405

Mark Elsen — Sat, 03 Aug 2019 17:19:54 GMT

- This looks to be a corrupted system, where upgrading may indeed not help, and leave the specific problem in place (shared lib not found). I would save the config , factory reset the controller and re-install 'everything'

Re: NTP broken on WLC 2405

Leo Laohoo — Sun, 04 Aug 2019 00:55:26 GMT

If you can afford a downtime, downgrade the firmware and upgrade again.
See if that fixes the issue.
What is the output to the command "sh time"?

Re: NTP broken on WLC 2405

dmcgrath.ca — Sun, 04 Aug 2019 02:14:06 GMT

@Mark Elsen wrote:

- This looks to be a corrupted system, where upgrading may indeed not help, and leave the specific problem in place (shared lib not found). I would save the config , factory reset the controller and re-install 'everything'
M.

Yeah, it does seem corrupted, doesn't it? My concern is how the system was able to pass any of its internal checks on the image if there was "tampering" (be it intentional, or corruption)?

AFAIK, the file system is Linux based, and similar to a standard initrd situation, where the root file system lives in a compressed file that gets loaded after the kernel boots, which is checksumed and packed into the kernel in the case of ARM, as I understand it. If this is the case, how could the security features of this design allow something like this to even boot, without failing an integrity check, let alone upgrading the image which should have replaced the entire kernel along with its packed initrd?

Re: NTP broken on WLC 2405

dmcgrath.ca — Sun, 04 Aug 2019 02:20:10 GMT

@Leo Laohoo wrote:
If you can afford a downtime, downgrade the firmware and upgrade again.
See if that fixes the issue.
What is the output to the command "sh time"?

Looks like it also triggers the missing shared lib:

(Cisco Controller) >show time/usr/bin/ntpq: error while loading shared libraries: libmd5.so.0: cannot open shared object file: No such file or directory


Time............................................. Sat Aug  3 22:18:27 2019

Timezone delta................................... 0:0
Timezone location................................ (GMT -5:00) Eastern Time (US and Canada)

NTP Servers
    NTP Version..................................     4

    Index  NTP Key       NTP Server   NTP Key   Polling Intervals
           Index                         Type        Max     Min
   -----------------------------------------------------------

 NTPQ status list of NTP associations

And of course, the time is still drifting, as expected.

I should be fine to take it down for a full wipe and reinstall. I have the means to dd or fully backup the CF card. Actually, I have another 1G CF card that I could just wipe and load up, and see if that helps. I just need to prepare the tftp server with the images and everything. Maybe later when nobody is using the WiFi I can give it a few shots.

What gets me though, is why an upgrade from 135 to 151 didn't fully replace the image and files? The shared libs the system uses should be included in the actual image, shouldn't it?

Re: NTP broken on WLC 2405

Leo Laohoo — Sun, 04 Aug 2019 04:10:35 GMT

What happens if NTP is configured?

Re: NTP broken on WLC 2405

Mark Elsen — Sun, 04 Aug 2019 08:52:48 GMT

>If this is the case, how could the security features of this design allow ...

- I wouldn't ponder to much about such issues; the only thing you can do is follow vendor-procedures to get things right.

Re: NTP broken on WLC 2405

HaifengLi — Sun, 04 Aug 2019 09:14:03 GMT

Hi,

The problem is here.

NTP Servers
    NTP Version..................................     4

Use the following command to see if there is any improvement.

config time ntp version 3

Best regards,

Haifeng

Re: NTP broken on WLC 2405

dmcgrath.ca — Sun, 04 Aug 2019 10:35:37 GMT

@HaifengLi wrote:

Hi,
The problem is here.
NTP Servers
    NTP Version..................................     4
Use the following command to see if there is any improvement.
config time ntp version 3
Best regards,
Haifeng

Well that appears to work just fine! Sounds almost like ntp version 4 does the typical `ntpq -p` but fails due to the missing shared lib, but ntp version 3 doesn't call the same file. Odd!

After setting NTP to version 3, and then putting in the server, I get:

(Cisco Controller) >show time

Time............................................. Sun Aug  4 06:31:34 2019

Timezone delta................................... 0:0
Timezone location................................ (GMT -5:00) Eastern Time (US and Canada)

NTP Servers
    NTP Version..................................     3
    NTP Polling Interval.........................     600

     Index     NTP Key Index                  NTP Server                Status          NTP Msg Auth Status
    -------  ---------------------------------------------------------------------
       1              0                                   10.1.1.1     In Sync              AUTH DISABLED

I'm not quite sure why 4 is so buggy, but much appreciate the help on that!

Cheers o/