SD-WAN lab, can not start NMS application server

[vpopov2]

HI  Team,

I had working SD-WAN lab in CML but can not access to vmanage anymore

request nms status all shows the fiollowingL
NMS application server
Enabled: false
Status: not running

It looks like it happens because because of  /opt/data disk is nearly full

Manager01:/opt/data/backup$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
none 16374944 4 16374940 1% /dev
/dev/sda1 5231612 1391268 3840344 27% /boot
/dev/loop0 1353216 1353216 0 100% /rootfs.ro
/dev/sda2 15596412 2310956 12430212 16% /rootfs.rw
aufs 15596412 2310956 12430212 16% /
tmpfs 16382116 1112 16381004 1% /run
shm 16382116 1368 16380748 1% /dev/shm
tmp 1048576 84 1048492 1% /tmp
/dev/sdb 51343840 43264632 5441384 89% /opt/data
/dev/sr0 372 372 0 100% /run/media/sr0
cgroup 16382116 0 16382116 0% /sys/fs/cgroup
svtmp 2048 1480 568 73% /etc/sv
Manager01:/opt/data/backup$

Is there a way to clean the data so I can restart the service ?  I do not have superuser  account to check the disk, should I open TAC case for help or is there any other way to solve it ?

I already tries to stop/start the lab a few times but it did not help

[bigevilbeard]

It looks like you have some big back up files, use ls -lh in your backup dir, you would look for the old or not used ones, obvs be careful which you delete here! Then restart the nms services.I dont think you can call TAC unless you are using the enterprise version. For CML-P, which i see you have done https://community.cisco.com/t5/cisco-modeling-labs-discussions/sd-wan-lab-can-not-start-nms-application-server/td-p/5337500

[vpopov2]

Thanks. In fact only the fime in backup folder I created after the issue started,
Manager01:~$ ls -l /opt/data/backup/
total 1576
-rwxrwxr-x 1 vmanage admin 1606723 Sep 26 14:56 bz-26-09-2026.tar.gz
drwxrwxrwx 2 vmanage vmanage 4096 Sep 26 14:56 staging

the issue started shortly after I  uploaded a new qcow2 image viptela-vmanage-20.12.4-genericx86-64.qcow2   So I wonder if there is a way to detete that file  somehow with only admin account?
Also is there a way to retrieve that backup from vmanage to external jumphost. One idea was to destroy that lab, recreate and restore db from backup. Thanks for your help so far

[bigevilbeard]

Arh ok ... so you might be able to ssh into the vmanage and remove the unwanted software old one? Is that request software delete-image, i cannot recall the command, the check your disk again for space

[vpopov2]

Thanks, it looks it was never properly installed , I only see the current version, I tried a few commands but it does not work, without superuser password  it will not possible to proceed ...

Manager01# show software

VERSION ACTIVE DEFAULT PREVIOUS CONFIRMED TIMESTAMP
----------------------------------------------------------------------------
20.12.4.1 true true false user 2025-02-07T12:07:48-00:00

Manager01# request software delete-image 20.12.6
status Could not find /images/20.12.6
Manager01#
Manager01# request software remove 20.12.6
Error: error-reason 'remove 20.12.6: Failed'
Manager01#

Manager01# request software remove 20.12.4.1
Error: error-reason 'remove 20.12.4.1: Failed'
Manager01# request software remove
Error: error-reason 'remove : Failed'
Manager01# request software remove 20.12.4.1
Error: error-reason 'remove 20.12.4.1: Failed'
Manager01# request software remove
Error: error-reason 'remove : Failed'

anager01:~$ cd /images/
bash: cd: /images/: Permission denied
Manager01:~$ ls -l /images
ls: cannot open directory '/images': Permission denied
Manager01:~$ exit
exit
Manager0

[bigevilbeard]

Hmm ok so this looks like the issue then, its not installed and sat in a temp dir and using up all your storage. See if you can remove this in vshell, i wonder if this gives you enough admin, you might want to back up here, in case something goes south!

[vpopov2]

This is the issue, I do not think I can  find and delete that files without superuser account (permission denied message for most of the folders). Also I have config db backup file but I  do not have ssh/sftp access to vmanage  , only console access from CML itself. So the question was how can I  retrieve that backup file from vManage file system to CML so I can store externally in my situation when I have  no https access to vmanage anymore. 

[bigevilbeard]

@vpopov2 i know some folks in the CML team, i just messaged them for advise here and how they can help.

[rkochery]

HI @vpopov2 , We recently came across issues with NMS services not coming up in vManage nodes, so far switching the default driver from ide to virtio seems to resolve the issue, could you please try the change in the vManage node definition? With virtio disk driver, during boot up, you will be prompted to select the storage device and please select vdb. 

[bigevilbeard]

Thanks @rkochery