Re: vManage server is not ready or temporarily unavailable

williancassiom19 · ‎08-03-2023

Hi everyone.

I have one vManage (20.7.1) in standalone mode in my eve-ng. Yesterday I had an eletric power disruption at home and my vManage seems to have died.

I can telnet to 8443 from my desktop. But I've tried to wget through vMware console and I got 503 error like below.

NMS services seems to be ok:

I performed the command: request nms configuration-db diagnostics ang I've got the output:

Is it the reason of failure? How to fix that? I have tried to start this service, but I couldn't.

Furthermore, I tried to make a copy of the vManage's DB using the following command, but it didn't create the .tar.gz file as described. It looks like to create only the log file about this tried.

I hope someone could help me out to get a backup file from vManage database (specially regarding the templates) either having the GUI interface access enabled again...

Thanks in advance.

bigevilbeard · ‎08-03-2023

Copied from another thread with the same issue

These are complicated issues to figure out given that many components come to play for services to come up online and operate as expected but you can do this to have a better idea of what the problem might be:

In vManage CLI: #request nms all restart
Now get into vshell mode: #vshell
On vshell mode: $ tail -f /var/log/nms/vmanage-server.log
Try to track any error message or something that looks obvious regarding application services
If you can log the session, upload it to review it as well.
You may want to have 2 CLI sessions opened one where you can run a "request nms all status" to check if services are coming up and the other one with the tail -f command.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io

williancassiom19 · ‎08-04-2023

Hi @bigevilbeard

I have collected the logs as mentioned and catched some error messages as below. The entire file was attached as well.

I am turned if you could have a light on this.

Much appreciate that.

Output:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.08.04 10:50:09 =~=~=~=~=~=~=~=~=~=~=~=

Successfully stopped NMS CloudAgent v2

[36mINFO[0m[2023-08-04T13:50:36.095666460Z] Container 20ec2e40c242d796cf69a749168a39c8e150c631dd2f5ae780bac46908734479 failed to exit within 10 seconds of signal 15 - using the force

[34mINFO[0m[2023-08-04T13:50:36.238595971Z] shim reaped [34mid[0m=20ec2e40c242d796cf69a749168a39c8e150c631dd2f5ae780bac46908734479

[36mINFO[0m[2023-08-04T13:50:36.248410291Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:50:36.304371643Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS cloud agent

Successfully stopped NMS service proxy

[36mINFO[0m[2023-08-04T13:50:47.413085395Z] Container b4b40c1e4cd5e62e499a5a12485a7fc8abec933e4e5b662e141bc5ae44ab989f failed to exit within 10 seconds of signal 15 - using the force

[34mINFO[0m[2023-08-04T13:50:47.523448367Z] shim reaped [34mid[0m=b4b40c1e4cd5e62e499a5a12485a7fc8abec933e4e5b662e141bc5ae44ab989f

[36mINFO[0m[2023-08-04T13:50:47.532715733Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:50:47.542607502Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS service proxy rate limit

[36mINFO[0m[2023-08-04T13:50:57.606185451Z] Container bf0cd46a5ccc989066629b409232f5b623917a62410479a94ccabf9d68802a61 failed to exit within 10 seconds of signal 15 - using the force

[34mINFO[0m[2023-08-04T13:50:57.721849922Z] shim reaped [34mid[0m=bf0cd46a5ccc989066629b409232f5b623917a62410479a94ccabf9d68802a61

[36mINFO[0m[2023-08-04T13:50:57.732761131Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:50:57.806564312Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS application server

Successfully stopped NMS data collection agent

Successfully stopped NMS messaging server

[34mINFO[0m[2023-08-04T13:51:04.840662284Z] shim reaped [34mid[0m=580717c371f2c4f587ce9289c194ac19aab37ae18d4c949c7303b334f56a85bd

[36mINFO[0m[2023-08-04T13:51:04.850367844Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:51:04.933727844Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS coordination server

[34mINFO[0m[2023-08-04T13:51:05.429794576Z] shim reaped [34mid[0m=a183621fc7fbf04e0574873cea992497fe55ea756d7abde303a226bc0a809a72

[36mINFO[0m[2023-08-04T13:51:05.439460458Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:51:05.512180646Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS configuration database

[34mINFO[0m[2023-08-04T13:51:11.132989089Z] shim reaped [34mid[0m=1b227f87593bb62f846b5c9aff489d49026df0278805c6fa5292c24d9e20de40

[36mINFO[0m[2023-08-04T13:51:11.142700488Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:51:11.271061927Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully stopped NMS statistics database

[34mINFO[0m[2023-08-04T13:51:11.847407844Z] shim reaped [34mid[0m=4770c866d4eb09c628b0806058ffbf2ffcf9b4115135ce00d4ac7f05afee56dd

[36mINFO[0m[2023-08-04T13:51:11.857164936Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:51:11.936462364Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:12.020346914Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:12.037121336Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS statistics database

[36mINFO[0m[2023-08-04T13:51:12.057738710Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:12.057796829Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[33mWARN[0m[2023-08-04T13:51:12.078846628Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:12.093553490Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS configuration database

[34mINFO[0m[2023-08-04T13:51:12.117852037Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/126fe1766ef8c59f7acec4514d5a7795410460e65a1499366d2b8606aab1e3cd/shim.sock" [34mdebug[0m=false [34mpid[0m=32285

[36mINFO[0m[2023-08-04T13:51:12.119800385Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:12.119866794Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[33mWARN[0m[2023-08-04T13:51:12.159581756Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:12.172182067Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS coordination server

[36mINFO[0m[2023-08-04T13:51:12.192645605Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:12.192701500Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[33mWARN[0m[2023-08-04T13:51:12.325464455Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:12.351792029Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS messaging server

[36mINFO[0m[2023-08-04T13:51:12.374487971Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:12.374540867Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[34mINFO[0m[2023-08-04T13:51:12.401409025Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/9eb91573fa47893d20fcd7a70ae8e2b125604dd7bf128a70051859fd259fedb7/shim.sock" [34mdebug[0m=false [34mpid[0m=32520

[34mINFO[0m[2023-08-04T13:51:12.608429671Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/ce759046ae19d75f977dc82cea80b17a6b314848a477e3d7268a17fb46089dec/shim.sock" [34mdebug[0m=false [34mpid[0m=32646

[34mINFO[0m[2023-08-04T13:51:12.624920138Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49/shim.sock" [34mdebug[0m=false [34mpid[0m=32658

Successfully started NMS application server

chown: cannot access '': No such file or directory

Successfully started NMS data collection agent

[33mWARN[0m[2023-08-04T13:51:13.433919877Z] Published ports are discarded when using host network mode

[33mWARN[0m[2023-08-04T13:51:13.435456618Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[34mINFO[0m[2023-08-04T13:51:13.442967882Z] shim reaped [34mid[0m=acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49

[36mINFO[0m[2023-08-04T13:51:13.453134950Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"

[33mWARN[0m[2023-08-04T13:51:13.457486909Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS service proxy

[33mWARN[0m[2023-08-04T13:51:13.479623201Z] Published ports are discarded when using host network mode

[34mINFO[0m[2023-08-04T13:51:13.503576140Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/dd9ad299a33bed387784eac6a29c60434ad67c4b4e4317b396ff6a2b5d59d591/shim.sock" [34mdebug[0m=false [34mpid[0m=770

[33mWARN[0m[2023-08-04T13:51:13.539837188Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[33mWARN[0m[2023-08-04T13:51:13.555180476Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS service proxy rate limit

[33mWARN[0m[2023-08-04T13:51:13.611192470Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[36mINFO[0m[2023-08-04T13:51:13.639023349Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:13.641678130Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[34mINFO[0m[2023-08-04T13:51:13.835574456Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/4814c8834f31aed684b46cf332bad899473ad8af9f1f96289c8b26a77ced4559/shim.sock" [34mdebug[0m=false [34mpid[0m=1061

python3: can't open file '/etc/sv/cloudagent/httpProxy/proxy.py': [Errno 2] No such file or directory

Successfully started NMS cloud agent

[33mWARN[0m[2023-08-04T13:51:14.235376682Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

Successfully started NMS CloudAgent v2

vManage# [33mWARN[0m[2023-08-04T13:51:14.251485326Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs

[36mINFO[0m[2023-08-04T13:51:14.269713412Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:14.269765318Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[34mINFO[0m[2023-08-04T13:51:14.330682508Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/32861389271f54e7e95f2be26b5f6de5ab5fbaf5cfa1dc632b573c3349f614f6/shim.sock" [34mdebug[0m=false [34mpid[0m=1235

[36mINFO[0m[2023-08-04T13:51:17.998941563Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]

[36mINFO[0m[2023-08-04T13:51:17.998995433Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

[34mINFO[0m[2023-08-04T13:51:18.116331354Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49/shim.sock" [34mdebug[0m=false [34mpid[0m=1442

./finish: line 29: kill: (3584) - No such process

Setting up watches.

Watches established.

vManage:~$ tail -f /var/log/nms/vmanage-server.log

04-Aug-2023 12:13:24,480 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-2-EventThread) || Authentication failed

[...] OUTPUT OMITTED

Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Could not perform discovery for database '<default database>'. No routing server available.

[...] OUTPUT OMITTED

Suppressed: org.neo4j.driver.exceptions.DiscoveryException: Failed to update routing table with server 'localhost(127.0.0.1):7687'.

[...] OUTPUT OMITTED

Caused by: org.neo4j.driver.exceptions.TransientException: Unable to get a routing table for database 'neo4j' because this database is unavailable

[...] OUTPUT OMITTED

Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause

[...] OUTPUT OMITTED

/var/log/nms/vmanage-server.log:04-Aug-2023 12:13:24,480 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-2-EventThread) || Authentication failed

[...] OUTPUT OMITTED

/var/log/nms/vmanage-server.log:04-Aug-2023 12:14:02,800 UTC ERROR [] [Application] (vManage-akka.actor.default-dispatcher-3) || Application failed to start

[...] OUTPUT OMITTED

/var/log/nms/vmanage-server.log:04-Aug-2023 13:50:52,691 UTC ERROR [vManage] [LoginRedirectFilter] (ServerService Thread Pool -- 87) || Filter class com.viptela.vmanage.server.auth.LoginRedirectFilter destroy called!!

[...] OUTPUT OMITTED

/var/log/nms/vmanage-server.log:04-Aug-2023 13:53:05,829 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-4-EventThread) || Authentication failed

[...] OUTPUT OMITTED

/var/log/nms/vmanage-server.log:04-Aug-2023 13:53:40,562 UTC ERROR [] [Application] (vManage-akka.actor.default-dispatcher-2) || Application failed to start

vManage:~$

bigevilbeard · ‎08-04-2023

Based on the txt file the error is auplink flush failed: "exec: \"auplink\": executable file not found in $PATH". This means that the auplink command could not be found in the system's $PATH environment variable. The $PATH variable is a list of directories that the shell searches for executable files.

The auplink command is used to flush the AUPlink cache. AUPlink is a tool that is used to manage network interfaces.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io

williancassiom19 · ‎08-15-2023

Hi @bigevilbeard, thanks for replying and so sorry about my delayed response.

To make a comparison, I build up a new vManage using the same eve-ng image and I saw that the new one has the same log status regarding 'auplink'.

Do know how can I fix the 'auplink' error?

Much thanks in advance,

BR,

bigevilbeard · ‎08-15-2023

Maybe try “sudo apt install auplink“ and once the auplink tool is installed, you need to add its path to the $PATH environment variable. To do this, open the .bashrc file in your home directory.

I’ve only every seen this issue with Docker before. This helped in the fix then.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io

williancassiom19 · ‎08-16-2023

The key point here is about I don't have the root password to do. Sadly, it couldn't be used even analyze the log error like below cause I have only the admin credentials.

bigevilbeard · ‎08-16-2023

I dont think you do this on the vManage but the VM this is running on. I would ask in the support at EVE if they have seen similar issue to this, as it looks like a docker issue.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io

ericgar · ‎08-18-2023

All 20.x versions don't have SU password, but token challenges for getting root privilege, only TAC can provide it. Try getting an admin-tech instead and see if neo4j logs are parte of the /var/log/ output bundle.

williancassiom19 · ‎08-18-2023

Hi @ericgar

I've got the admin tech as you mentioned. I am attaching the file if you (or someone else) could help me.

If I could restore the templates from the old vManage, it will be very helpfull.

Thanks in advance,