Hi,
I don't see exact config of NTP on router and server, trying general troubleshooting steps just re-check config before debugging.
Ensure NTP version/server IP/ authentication parameters (if any) match, use "prefer" keyword. Try to change time on router manually the same as in server and let them sync. again.
Better share config and "show ntp peer" output.
HTH,
HTH,
Please rate and mark as an accepted solution if you have found any of the information provided useful.