Hi my friend.
Coud you help me out?
I have one vManage (20.7.1) in standalone mode in my eve-ng. Yesterday I had an eletric power disruption at home and my vManage seems to have died.
I can telnet to 8443 from my desktop. But I've tried to wget through vMware console and I got 503 error like below.
NMS services seems to be ok:
I performed the command: request nms configuration-db diagnostics ang I've got the output:
Is it the reason of failure? How to fix that? I have tried to start this service, but I couldn't.
Furthermore, I tried to make a copy of the vManage's DB using the following command, but it didn't create the .tar.gz file as described. It looks like to create only the log file about this tried.
I have collected the logs and catched some error messages as below. The entire file was attached as well.
I am turned if you could have a light on this.
Much appreciate that.
Output:
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.08.04 10:50:09 =~=~=~=~=~=~=~=~=~=~=~=
Successfully stopped NMS CloudAgent v2
[36mINFO[0m[2023-08-04T13:50:36.095666460Z] Container 20ec2e40c242d796cf69a749168a39c8e150c631dd2f5ae780bac46908734479 failed to exit within 10 seconds of signal 15 - using the force
[34mINFO[0m[2023-08-04T13:50:36.238595971Z] shim reaped [34mid[0m=20ec2e40c242d796cf69a749168a39c8e150c631dd2f5ae780bac46908734479
[36mINFO[0m[2023-08-04T13:50:36.248410291Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:50:36.304371643Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS cloud agent
Successfully stopped NMS service proxy
[36mINFO[0m[2023-08-04T13:50:47.413085395Z] Container b4b40c1e4cd5e62e499a5a12485a7fc8abec933e4e5b662e141bc5ae44ab989f failed to exit within 10 seconds of signal 15 - using the force
[34mINFO[0m[2023-08-04T13:50:47.523448367Z] shim reaped [34mid[0m=b4b40c1e4cd5e62e499a5a12485a7fc8abec933e4e5b662e141bc5ae44ab989f
[36mINFO[0m[2023-08-04T13:50:47.532715733Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:50:47.542607502Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS service proxy rate limit
[36mINFO[0m[2023-08-04T13:50:57.606185451Z] Container bf0cd46a5ccc989066629b409232f5b623917a62410479a94ccabf9d68802a61 failed to exit within 10 seconds of signal 15 - using the force
[34mINFO[0m[2023-08-04T13:50:57.721849922Z] shim reaped [34mid[0m=bf0cd46a5ccc989066629b409232f5b623917a62410479a94ccabf9d68802a61
[36mINFO[0m[2023-08-04T13:50:57.732761131Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:50:57.806564312Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS application server
Successfully stopped NMS data collection agent
Successfully stopped NMS messaging server
[34mINFO[0m[2023-08-04T13:51:04.840662284Z] shim reaped [34mid[0m=580717c371f2c4f587ce9289c194ac19aab37ae18d4c949c7303b334f56a85bd
[36mINFO[0m[2023-08-04T13:51:04.850367844Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:51:04.933727844Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS coordination server
[34mINFO[0m[2023-08-04T13:51:05.429794576Z] shim reaped [34mid[0m=a183621fc7fbf04e0574873cea992497fe55ea756d7abde303a226bc0a809a72
[36mINFO[0m[2023-08-04T13:51:05.439460458Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:51:05.512180646Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS configuration database
[34mINFO[0m[2023-08-04T13:51:11.132989089Z] shim reaped [34mid[0m=1b227f87593bb62f846b5c9aff489d49026df0278805c6fa5292c24d9e20de40
[36mINFO[0m[2023-08-04T13:51:11.142700488Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:51:11.271061927Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully stopped NMS statistics database
[34mINFO[0m[2023-08-04T13:51:11.847407844Z] shim reaped [34mid[0m=4770c866d4eb09c628b0806058ffbf2ffcf9b4115135ce00d4ac7f05afee56dd
[36mINFO[0m[2023-08-04T13:51:11.857164936Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:51:11.936462364Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:12.020346914Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:12.037121336Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS statistics database
[36mINFO[0m[2023-08-04T13:51:12.057738710Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:12.057796829Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[33mWARN[0m[2023-08-04T13:51:12.078846628Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:12.093553490Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS configuration database
[34mINFO[0m[2023-08-04T13:51:12.117852037Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/126fe1766ef8c59f7acec4514d5a7795410460e65a1499366d2b8606aab1e3cd/shim.sock" [34mdebug[0m=false [34mpid[0m=32285
[36mINFO[0m[2023-08-04T13:51:12.119800385Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:12.119866794Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[33mWARN[0m[2023-08-04T13:51:12.159581756Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:12.172182067Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS coordination server
[36mINFO[0m[2023-08-04T13:51:12.192645605Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:12.192701500Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[33mWARN[0m[2023-08-04T13:51:12.325464455Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:12.351792029Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS messaging server
[36mINFO[0m[2023-08-04T13:51:12.374487971Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:12.374540867Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[34mINFO[0m[2023-08-04T13:51:12.401409025Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/9eb91573fa47893d20fcd7a70ae8e2b125604dd7bf128a70051859fd259fedb7/shim.sock" [34mdebug[0m=false [34mpid[0m=32520
[34mINFO[0m[2023-08-04T13:51:12.608429671Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/ce759046ae19d75f977dc82cea80b17a6b314848a477e3d7268a17fb46089dec/shim.sock" [34mdebug[0m=false [34mpid[0m=32646
[34mINFO[0m[2023-08-04T13:51:12.624920138Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49/shim.sock" [34mdebug[0m=false [34mpid[0m=32658
Successfully started NMS application server
chown: cannot access '': No such file or directory
Successfully started NMS data collection agent
[33mWARN[0m[2023-08-04T13:51:13.433919877Z] Published ports are discarded when using host network mode
[33mWARN[0m[2023-08-04T13:51:13.435456618Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[34mINFO[0m[2023-08-04T13:51:13.442967882Z] shim reaped [34mid[0m=acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49
[36mINFO[0m[2023-08-04T13:51:13.453134950Z] ignoring event [36mmodule[0m=libcontainerd [36mnamespace[0m=moby [36mtopic[0m=/tasks/delete [36mtype[0m="*events.TaskDelete"
[33mWARN[0m[2023-08-04T13:51:13.457486909Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS service proxy
[33mWARN[0m[2023-08-04T13:51:13.479623201Z] Published ports are discarded when using host network mode
[34mINFO[0m[2023-08-04T13:51:13.503576140Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/dd9ad299a33bed387784eac6a29c60434ad67c4b4e4317b396ff6a2b5d59d591/shim.sock" [34mdebug[0m=false [34mpid[0m=770
[33mWARN[0m[2023-08-04T13:51:13.539837188Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[33mWARN[0m[2023-08-04T13:51:13.555180476Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS service proxy rate limit
[33mWARN[0m[2023-08-04T13:51:13.611192470Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[36mINFO[0m[2023-08-04T13:51:13.639023349Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:13.641678130Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[34mINFO[0m[2023-08-04T13:51:13.835574456Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/4814c8834f31aed684b46cf332bad899473ad8af9f1f96289c8b26a77ced4559/shim.sock" [34mdebug[0m=false [34mpid[0m=1061
python3: can't open file '/etc/sv/cloudagent/httpProxy/proxy.py': [Errno 2] No such file or directory
Successfully started NMS cloud agent
[33mWARN[0m[2023-08-04T13:51:14.235376682Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
Successfully started NMS CloudAgent v2
vManage# [33mWARN[0m[2023-08-04T13:51:14.251485326Z] auplink flush failed: [33merror[0m="exec: \"auplink\": executable file not found in $PATH" [33mmethod[0m=Unmount [33mstorage-driver[0m=aufs
[36mINFO[0m[2023-08-04T13:51:14.269713412Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:14.269765318Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[34mINFO[0m[2023-08-04T13:51:14.330682508Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/32861389271f54e7e95f2be26b5f6de5ab5fbaf5cfa1dc632b573c3349f614f6/shim.sock" [34mdebug[0m=false [34mpid[0m=1235
[36mINFO[0m[2023-08-04T13:51:17.998941563Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
[36mINFO[0m[2023-08-04T13:51:17.998995433Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
[34mINFO[0m[2023-08-04T13:51:18.116331354Z] shim containerd-shim started [34maddress[0m="/containerd-shim/moby/acef1fa256795ac822f681a0086de37a1ef52da1bce19518b96627e159d3cd49/shim.sock" [34mdebug[0m=false [34mpid[0m=1442
./finish: line 29: kill: (3584) - No such process
Setting up watches.
Watches established.
vManage:~$ tail -f /var/log/nms/vmanage-server.log
04-Aug-2023 12:13:24,480 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-2-EventThread) || Authentication failed
[...] OUTPUT OMITTED
Caused by: org.neo4j.driver.exceptions.ServiceUnavailableException: Could not perform discovery for database '<default database>'. No routing server available.
[...] OUTPUT OMITTED
Suppressed: org.neo4j.driver.exceptions.DiscoveryException: Failed to update routing table with server 'localhost(127.0.0.1):7687'.
[...] OUTPUT OMITTED
Caused by: org.neo4j.driver.exceptions.TransientException: Unable to get a routing table for database 'neo4j' because this database is unavailable
[...] OUTPUT OMITTED
Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause
[...] OUTPUT OMITTED
/var/log/nms/vmanage-server.log:04-Aug-2023 12:13:24,480 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-2-EventThread) || Authentication failed
[...] OUTPUT OMITTED
/var/log/nms/vmanage-server.log:04-Aug-2023 12:14:02,800 UTC ERROR [] [Application] (vManage-akka.actor.default-dispatcher-3) || Application failed to start
[...] OUTPUT OMITTED
/var/log/nms/vmanage-server.log:04-Aug-2023 13:50:52,691 UTC ERROR [vManage] [LoginRedirectFilter] (ServerService Thread Pool -- 87) || Filter class com.viptela.vmanage.server.auth.LoginRedirectFilter destroy called!!
[...] OUTPUT OMITTED
/var/log/nms/vmanage-server.log:04-Aug-2023 13:53:05,829 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-4-EventThread) || Authentication failed
[...] OUTPUT OMITTED
/var/log/nms/vmanage-server.log:04-Aug-2023 13:53:40,562 UTC ERROR [] [Application] (vManage-akka.actor.default-dispatcher-2) || Application failed to start
vManage:~$