Summary: Data & Assurance Backup for Cisco DNAC setup.
Issue encountered: Backup failed with below error.
Task error description: Error during _process_backup(): Internal server error: {"error":{"root_cause":[{"type":"snapshot_creation_exception","reason":"[ndp:********* failed to create snapshot"}],"type":"snapshot_creation_exception","reason":"[ndp:******** failed to create snapshot","caused_by":{"type":"access_denied_exception","reason":"/var/data/es/snapshots/meta-UU-******.dat"}},"status":500}
Solution:
1. Do not use NetApp NAS Storage solution as a NFS server. This will create additional hops and performance challenges.
2. Go with Local File System present on UNIX machine. You can choose supported Linux OS as a host.
Example - These IP addresses, mount point names, IDs are not real. They are imaginary ones.
10.10.10.10 is the DNAC System
10.10.10.20 is the Linux Host 1.
20.20.20.20 is the Linux Host 2.
For Data Backup, I am using Linux Host 2.
For Assurance Backup (Data + Snapshot entry), I am using Linux Host 1. Try to keep Linux Host 1 and Cisco DNAC IP in same Network (if possible) or at-least avoid NFS firewall blockage as much as possible.
You can also go with one single Linux Host (for Data backup FS and Assurance Backup FS) if you are confident that there is no NFS port firewall blockage between Cisco DNAC system and Target UNIX server.
Note - Initially, I went with one single host (Host 2 in my case) - Data Backup was successful. Assurance Backup kept on failing with snapshot creation error. Later, I decided to go with two separate hosts.
Key NFS ports to be opened between Linux Host 1, 2 and Cisco DNAC system - 111, 635, 2049, 20048, 32767.
For Data Backup - created a File System named /DNAC_Backup with 500GB on Host 2.
For Assurance Backup - created a File System named /DNAC with 3TB on Host 1
Note - Size of FS completely depends on your environment and an amount of data to be backed up.
Permission and ownership on these 2 FS is very important. I ran below on both of these FS.
Linux Host 2: chmod -R 777 /DNAC_Backup
Linux Host 1: chmod -R 777 /DNAC
Linux Host 2 for Data Backup: For the purpose of testing - I ended up creating local ID abc123 on Host 2. You can also go with a LDAP ID. After this, changed ownership to below -
chown abc123:abc123 /DNAC_Backup
Linux Host 1 for Assurance set Backup: I went with ID nfsnobody or nobody.
chown nobody:nobody /DNAC
/DNAC for Assurance backup should also be exported to Cisco DNAC system.
Linux Host 1: 10.10.10.20:/root>cat /etc/exports
/DNAC <DNAC HOST IP of your environment - example 10.10.10.10>(rw,all_squash,sync,no_subtree_check)
Also, run showmount -e to make sure that export is working fine. I ran it on Linux Host 1 and it returned back DNAC system IP.
Note - After changing permission, ownership, creating exports, do not forget to restart NFS server.
Linux Host 1 and/or even 2:
systemctl start nfs-server
or,
sudo systemctl start nfs-server
Now, work on Cisco DNAC Backup Configuration setup.
Backup over Host / First Tab in Backup configuration wizard - I am considering this for Data Backup.
Target Host IP: 20.20.20.20
Post: 22
Path: /DNAC_Backup
ID: abc123
Password: ****
Encryption: **** (Give anything and remember this)
Click on Apply. Wait until you get Configured message. If it says, not configured, then you need to work with-
firewall teams to get NFS firewall ports opened.
UNIX support to work on permission and ownership modification of /DNAC_Backup FS.
Next comes,
Backup over NFS Server Tab / 2nd Backup configuration wizard - I am considering this for Assurance Backup.
Target Host IP: 10.10.10.20
Path: /DNAC
Click on Apply. Wait until you get Configured message. If it says, not configured, then you need to work with-
firewall teams to get NFS firewall ports opened and UNIX team to check permission, ownership and exports.
For troubleshooting, you can -
restart NFS server on Linux Host
recheck NFS export setup on Linux Host
recheck ownership and permission. Better to keep 777 and nobody:nobody against /DNAC FS
Try to remove entry from Cisco DNAC GUI wizard. Close the session. Login back again. Enter it again and then click on apply.
Once it says configured, you can also verify by opening CLI console of Cisco DNAC.
10.10.10.10:: df -h
This will return all FS present here. There will be one entry which says -
10.10.10.20:/DNAC /data/nfs
Once this is configured, it will appear over a mount point name /data/nfs.
Lastly, trigger backup. All data or without assurance data. I was more interested to see the status of all data (capturing assurance part). It started and then I noticed it created folders inside /DNAC.
/DNAC/ndp/es/snapshots. If you see entries started coming in to snapshots folder, this means backup started and will surely move ahead.
Finally, after running for 6 hours, it completed successfully and I could see indices and ******.dat critical backed up files.
Additional changes made: when I saw /ndp/es/snapshots folder got created inside /DNAC location.
chmod -R 777 /DNAC
Once again, I ran this as I noticed ndp, es, snapshots, indices folder got created with 755 permission. Rather then taking any risk, I went ahead and reapplied 777 full permission that too recursively.
While Backup runs -
Ensure to have stable network connectivity
Do no disconnect mounts, exports on Linux Host.
Key things to remember -
Do not give up.
Restart NFS server
Apply 777 Recursive permission to FS
Remove entries. Close DNAC session. Open it again. Enter values again.
All the Best.
Regards,
Obaidullah Salam (Obaid)
