06-05-2018 04:45 AM - edited 03-08-2019 03:15 PM
Hi Guys,
I am trying to upgrade to NDB embedded 3.6 on a Nexus 3538. When I try to run the activation script (python script) then it fails with 'no space on bootflash'. I have 850m free, how much space is actually needed? I have deleted as much as I can, and cannot delete anything more.
Thanks!
Jacques
Solved! Go to Solution.
06-07-2018 04:38 AM
Hi Jacques,
I am sorry you are still facing these issues :-( All points toward insufficient privileges of the account you are using to access the switch.
Please let me ask you: I know you are using the "admin" account - but is this account truly a local account? Additionally, what role is this account assigned to? We need it to be network-admin; network-operator is not enough. Verify this using show user-account.
Thank you!
Best regards,
Peter
06-05-2018 05:49 AM - edited 06-05-2018 05:50 AM
Hello Jacques,
The activator script only creates a tiny text file - there should be no large consumption of a filesystem space required at all.
May I ask you to post the complete transcript of your terminal/SSH session as you try to execute the activator script? Also, from the bash shell on your switch, can you post the output of
mount df df -i
Thank you!
Best regards,
Peter
06-05-2018 06:22 AM
Hi Peter,
I am running the python script from within bash itself. I have seen it in NX-OS as well, should I rather run it from there?
Then in the bash shell it complains about space on bootflash and that I should clean it. Pretty much this gets triggered in the script:
try:
with open('/bootflash/interfaces', 'w') as fd:
fd.write(data)
except IOError as e:
logger.error("No space left on the device")
logger.error("Pls clean up the /bootflash and retry")
sys.exit()
1 have about 850Mb free when executing it. But in the meantime I managed to upgrade to 3.0 because it does not require the script to run. I will attempt to do 3.6 again tomorrow with all the output you requested. but from within bash:
bash-3.2$ mount df df -i
mount: mount point df does not exist
bash-3.2$
I am running version 6.0(2)A8(6) with 2G bootflash and 4G RAM.
Jacques
06-05-2018 07:00 AM
Hi Jacques,
Two things:
First, it seems like the commands you have entered got garbled into one line. They are three independent commands, and need to be executed one after another:
mount
df
df -iSecond, are you accessing the switch using the admin username? If not, try logging in as admin and running the script. The error might very well be caused by the script not having enough privileges to create the /bootflash/interfaces file; the IOError exception is not specific.
Best regards,
Peter
06-05-2018 07:08 AM
Hi Peter,
Sorry about that, see below. I sign into the switch as admin/admin. when I do run bash, it only asks for a password when doing sudo su. Password is blank though, not sure why.
bash-3.2$ mount
/dev/root on / type auto (rw,errors=remount-ro)
proc on /proc type proc (rw)
none on /post type tmpfs (rw,size=2m,mode=777)
none on /var type tmpfs (rw,size=2m,mode=777)
none on /sys type sysfs (rw)
nodev on /debugfs type debugfs (rw)
none on /isan type tmpfs (rw,size=1500m,mode=755)
none on /nxos/tmp type tmpfs (rw,size=40m,mode=777)
none on /var/tmp type tmpfs (rw,size=400m,mode=777)
none on /var/sysmgr type tmpfs (rw,size=900m,mode=777)
none on /var/sysmgr/ftp type tmpfs (rw,size=300m,mode=777)
none on /var/sysmgr/ftp/cores type tmpfs (rw,size=100m,mode=777)
none on /dev/shm type tmpfs (rw,size=628m,mode=755)
none on /volatile type tmpfs (rw,size=100m,mode=777)
none on /debug type tmpfs (rw,size=20m,mode=777)
none on /dev/mqueue type mqueue (rw)
/dev/sda5 on /mnt/cfg/0 type ext3 (rw,nodev,noatime,nodiratime,noacl,data=writeback,commit=15,nouser_xattr,barrier=0,nocheck)
/dev/sda6 on /mnt/cfg/1 type ext3 (rw,nodev,noatime,nodiratime,noacl,data=writeback,commit=15,nouser_xattr,barrier=0,nocheck)
none on /var/sysmgr/startup-cfg type tmpfs (rw,size=400m,mode=777)
devpts on /dev/pts type devpts (rw,noexec,nosuid,newinstance,gid=4,mode=0600,ptmxmode=0666)
/dev/sda4 on /mnt/pss type ext3 (rw,nodev,noatime,nodiratime,noacl,data=writeback,commit=15,nouser_xattr,barrier=0,nocheck)
/dev/mtdblock2 on /mnt/plog type jffs2 (rw,noexec,nodev,noatime)
/dev/sda3 on /bootflash type ext3 (rw,noexec,nodev,noatime,nodiratime,noacl,data=writeback,commit=15,nouser_xattr,barrier=0,nocheck)
/dev/sdb1 on /usbslot1 type vfat (rw,noexec,nosuid,nodev,noatime,quiet,uid=2002,gid=503,umask=0,iocharset=default)
smackfs on /smack type smackfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
vdccontrol on /cgroup type cgroup (rw,cpu,memory,devices,cpuacct,cpuset)
bash-3.2$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 409600 77232 332368 19% /
none 2048 4 2044 1% /post
none 409600 77232 332368 19% /var
none 1536000 872692 663308 57% /isan
none 40960 128 40832 1% /nxos/tmp
none 409600 856 408744 1% /var/tmp
none 921600 96 921504 1% /var/sysmgr
none 307200 108 307092 1% /var/sysmgr/ftp
none 102400 0 102400 0% /var/sysmgr/ftp/cores
none 643072 170332 472740 27% /dev/shm
none 102400 0 102400 0% /volatile
none 20480 0 20480 0% /debug
/dev/sda5 38260 4351 31934 12% /mnt/cfg/0
/dev/sda6 37755 4351 31455 13% /mnt/cfg/1
none 409600 3952 405648 1% /var/sysmgr/startup-cfg
/dev/sda4 38268 6591 29701 19% /mnt/pss
/dev/mtdblock2 49024 1616 47408 4% /mnt/plog
/dev/sda3 1850752 1043064 713672 60% /bootflash
/dev/sdb1 3922944 1179832 2743112 31% /usbslot1
bash-3.2$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 475731 2736 472995 1% /
none 475731 2 475729 1% /post
none 475731 2736 472995 1% /var
none 475731 13204 462527 3% /isan
none 475731 8 475723 1% /nxos/tmp
none 475731 82 475649 1% /var/tmp
none 475731 27 475704 1% /var/sysmgr
none 475731 13 475718 1% /var/sysmgr/ftp
none 475731 1 475730 1% /var/sysmgr/ftp/cores
none 475731 290 475441 1% /dev/shm
none 475731 1 475730 1% /volatile
none 475731 1 475730 1% /debug
/dev/sda5 9880 43 9837 1% /mnt/cfg/0
/dev/sda6 9760 43 9717 1% /mnt/cfg/1
none 475731 582 475149 1% /var/sysmgr/startup-cfg
/dev/sda4 9880 139 9741 2% /mnt/pss
/dev/mtdblock2 0 0 0 - /mnt/plog
/dev/sda3 235200 78 235122 1% /bootflash
/dev/sdb1 0 0 0 - /usbslot1
06-05-2018 10:28 AM
Hi Jacques,
This is really strange. Let's dig deeper if you don't mind. Can you please execute these commands in succession and post the complete output?
run bash sudo su whoami groups ls -ald /bootflash ls -al /bootflash <run the NDB activator script here> exit exit
Thank you!
Best regards,
Peter
06-06-2018 01:30 AM
06-06-2018 10:12 PM
Hi Jacques,
I guess I see the problem:
-rw-r--r-- 1 2004 network-operator 126 Jun 4 10:57 interfaces
There already is a file named "interfaces" in your /bootflash but it is owned by an UID (User ID) that no longer exists as a valid user (it certainly isn't "admin"). Since only this user has the write privileges for this file while the group and others only have read privileges, an attempt to open this file by "admin" for write operations necessarily fails. In this sense, the activator script misleads us: For every I/O exception, it just summarily says that the filesystem is full - which may not be the case at all.
Here's my suggestion: Under bash, rename this file (I don't want to delete it, just in case):
run bash sudo su mv /bootflash/interfaces /bootflash/interfaces.old
Then try running the activator script again.
Best regards,
Peter
06-07-2018 01:10 AM
Hi Peter, unfortunately different error now. Looks like I do not have permissions to create a folder:
DC-MON-SW1# run bash
bash-3.2$ sudo su
Password:
bash-3.2$ cd /bootflash/
bash-3.2$ python ndbactivator3.py -v embndb
mkdir: cannot create directory `embndb': Permission denied
Traceback (most recent call last):
File "ndbactivator3.py", line 275, in <module>
main()
File "ndbactivator3.py", line 267, in main
get_iface_info(options.vservice_name,forceFlag)
File "ndbactivator3.py", line 111, in get_iface_info
os.chdir(cont_path+ '/' + dir_name)
OSError: [Errno 2] No such file or directory: '/isan/vdc_1/virtual-instance/embndb/rootfs/embndb'
bash-3.2$ cd /isan/vdc_1/virtual-instance/
bash-3.2$ dir
embndb ofa_ndbemb
bash-3.2$ cd embndb
bash-3.2$ dir
cisco rootfs
bash-3.2$ cd rootfs
bash-3.2$ dir
bin cisco etc lib lost+found mnt sbin tmp var
boot dev home linuxrc media proc sys usr xnclite
bash-3.2$ exit
jacques
06-07-2018 01:37 AM
Hi Jacques,
It appears that we're constantly fighting problems with permissions. It looks to me that your "admin" account, or whatever account it is that you use to log into the device, is a RADIUS/TACACS+ account, which may result in different UIDs between different logins.
Ideally, we should be using the "admin" account that is created locally and that uses a constant UID.
Here is my suggestion:
During this process, do not log off - make sure that all these steps are executed in a single terminal session. This will make sure that all the files will be created using your current UID which won't change during the installation. Hopefully, this will make it work.
Best regards,
Peter
06-07-2018 03:48 AM
Hi Peter, unfortunately same result. I uninstalled ndb and openflow. Removed interfaces file. Reinstalled ndb and openflow. Activated both and then:
DC-MON-SW1# run bash
bash-3.2$
bash-3.2$ sudo su
Password:
bash-3.2$ cd /bootflash/
bash-3.2$ python ndbactivator3.py -v ndb
mkdir: cannot create directory `embndb': Permission denied
Traceback (most recent call last):
File "ndbactivator3.py", line 275, in <module>
main()
File "ndbactivator3.py", line 267, in main
get_iface_info(options.vservice_name,forceFlag)
File "ndbactivator3.py", line 111, in get_iface_info
os.chdir(cont_path+ '/' + dir_name)
OSError: [Errno 2] No such file or directory: '/isan/vdc_1/virtual-instance/ndb/rootfs/embndb'
06-07-2018 04:38 AM
Hi Jacques,
I am sorry you are still facing these issues :-( All points toward insufficient privileges of the account you are using to access the switch.
Please let me ask you: I know you are using the "admin" account - but is this account truly a local account? Additionally, what role is this account assigned to? We need it to be network-admin; network-operator is not enough. Verify this using show user-account.
Thank you!
Best regards,
Peter
06-07-2018 04:58 AM
Hi Peter, I am glad to say I got it sorted out!
For some reason, when I started bash, and do sudo su and it asked for a password, a blank password did not report a authentication failure. Instead it appears that is was accepted. However I realized whoami still reported the user as admin. And the permissions on the folders all had root listed. So I could not even manually create folders there (anywhere for that matter). So I saw somewhere 'sudo su root' in general Linux forums and when I entered that together with the admin password, whoami reported root as the user. So then the script run without any issues.
But thanks for your help it really pointed me in the right direction!
Jacques
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide