Background :
As a system administrator, you want to plan for potential hardware or VNF software failures. After a service chain is deployed, you can create a backup.
In the event of ENCS hardware failure, Backup/Restore procedure can be used for restoring configuration on the Replacement device. Refer to FAQ 3 and 4 for some relevant information. When recovering configuration on a replacement device, ensure that the same hardware, software versions and images are uploaded and registered before using Restore.
Hardware and Software versions used
ENCS 54xx hardware
NFVIS 4.2.1FC3 or above
Prerequisites
NFVIS 4.2.1FC3 installed on hardware with access to the on-box web GUI.
VNF Service chain deployed and in active state.
Note: If networks are created but VNFs are NOT deployed and backup is attempted, backup file will not contain any VNF related info configuration. Sometimes, this option is used for optimizing process of setting up multiple devices.
Step-by-Step Instructions
Based on the Backup option chosen(choices in step 2), restore can be executed.
Note: Backup and Restore progress notification is being improved in the upcoming releases.
- Login to designated ENCS using Google chromeCreate a backup of deployed service chain.
- There are multiple levels of Backup possible. NFVIS Config ONLY / NFVIS config+ALL VNF virtual disk / Select VNF disk. You can select either option but one at a time.
- NFVIS Config Only / NFVIS config+ALL VNF virtual disk backup
NFVIS Config Only or System level(nfvis+vnf disk) Backup
- Select VNF virtual disk. This option would be selected for subsequent VNF specific restore.
Select VNF Export for subsequent Import
3. Restore from Backup
- Select a single VNF to import and restore
- Select and delete the specific VNF in the service chain. This is to simulate the condition where the VNF becomes unavailable
- Restore by selecting specific VNF. Picture below shows import of a single VNF.
Delete VNF and Import from proven VNF
- Restore NFVIS config-only or NFVIS config with ALL VNFs
Note1 : When restoring ALL VNFs, there is an option of restoring deployed VNFs without impacting the management connectivity, this is important when dealing with remote sites.
Note 2 : management connectivity is intact when restoring individual VNFs.
Frequently Asked Questions
1. During restore, is having the same platform a best practice or is it enforced in software?
It is software enforced. Different platforms have different architectures. Not all features are supported for all platforms. Ex.: Dual wan and DPDK are only supported for ENCS 5400. A backup from ENCS 5400 should not be restored to UCS.
2. If checked in software, what exactly is checked for, and how strict is it?
It is a strict check for PID. (nfvis# show platform-details hardware_info PID)
3. Is there any part of the config that is specific to the device?
Yes, SNMPv3 config as an example uses keys that are unique to the device. Due to known caveat until 3.12.3FC4, when snmpv3 and vbranch backup are planned, recommendation is to execute backup prior to snmpv3 configuration.
In the device where restored, snmpv3 must be configured.
4. How is VM licensing handled in restored VNFs?
Until NFVIS 3.12.3FC4, when NFVIS configuration is restored from backup, there is no option to retain the UUID of the VNF. As a result, the VNF serial number is expected to change. In the case of ISRv, a new token is used up.
Refer to ISRv smart license documentation for releasing token associated with deprecated ISRv.
Note : In NFVIS roadmap, there is an option to retain UUID, mac address thats considered. This will allow reuse of smart license in RMA use-case.
5. What would be the steps for RMA on ENCS device?
a. Stage the RMA box in the lab so you have can the console and Mgmt access
b. Install the NFVIS version used in the device deployed device
c. If using images prior to 4.2.1FC3, VNF images need to be uploaded. NFVIS4.2.1FC3 or higher, skip this step.
d. Upload the system/vm backup files, restore it. ISRv smart license is recovered as part of VNF restore.
e. If using NFVIS earlier than 4.2.1FC3, Install the smart license, and re-generate the certificates on the ISRv
6. Could you restore a 5406 to a 5408?
No, we cannot restore a backup from ENCS 5406 to a ENCS 5408. If we add a new feature specific to ENCS 5408 in future, we do not want it to conflict with ENCS 5408
7. Does restored NFVIS need to have the exact same amount of memory, disks, etc?
No. As of now, we do not check for disk space. Also, there is no need for restored NFVIS to have exact same amount of memory, disks. However, if any VM is deployed on external disk, restored NFVIS expects an external disk in the same slot to redeploy that VM
- Please elaborate on what is meant by "VM configurations”.
From the context of conf-only backup, VM configurations are NFVIS related VM configurations. In deeper technical terms, it means domain-xml for kvm deployments which included number of CPU and memory required, disk-space, I/O controllers, etc. From conf-and-vms backup, as the VM disks are also backed up, all configurations including configurations within VMs are backed up
8. "If a VM is in an invalid state.." – according to what?
nfvis# show vm_lifecycle opdata tenants tenant admin deployment <deployment-name>
shows current state of the VM. A VM can be in following states: UNDEFINED_STATE, DEPLOYING_STATE, MONITOR_UNSET_STATE (valid state), MONITOR_DISABLED_STATE, STOPPING_STATE, SHUTDOWN_STATE (valid state), STARTING_STATE, REBOOTING_STATE, INERT_STATE (valid state for unmonitored VMs), ALIVE_STATE (valid state), UNDEPLOYING_STATE, ERROR_STATE (valid state)
- Explain the format of the backup file - what files are in it and how you can view contents of it?
The backup package is a gzipped tarball. It can be inflated using any tar utility. It contains config.xml, manifest file and list of VM export files with their SHA1 checksums
- About how long does a backup take?
The answer to this question would be a relative answer. There is no time bound for backup. It depends on type of backup, size of VM disks, number of VM disks, and storage used for backup. A simple conf-only backup to intdatastore completes within seconds. A conf-and-vms backup with multiple large VM disks to a USB takes 9-10 hours.
9. How/where do you download the backup file via GUI? CLI?
The portal shows a list of backup files available on system on Backup-Restore page (Host à Utilities à Backup - Restore). It can be downloaded from there. For CLI, nfvis# show system file-list shows backup files. These can be SCP’ed out of NFVIS or copied to USB drive if disk space permits it.
10. What logs capture backup progress, where should we start with troubleshooting if a problem is encountered?
All logs for backup and restore are logged in nfvis_config.log. As usual, debug logs show a detailed information and proves helpful in troubleshooting
- Any way to run scheduled backups?
No. As of now, there is no way of doing this. - How "clean" does a system have to be to run a restore? Can you run it on a fully configured system with running VMs, etc? If so, how does config get merged?
As long as confd, esc-lite and nfvos-confd daemons are running, a restore can be performed. Yes, it can be performed on fully configured system. In this case, all previous configurations are erased and overwritten by configurations from backup package. (Except connectivity settings when ‘except-connectivity’ option is used)
11. How long does it take to run a restore once the backup is uploaded?
This is also relative question. Restore is also not time bound. It depends on the time taken to deploy VMs, number of VMs deployed, storage option used.
- Can you restore on to same platform with a different IP/hostname?
Yes. ’except-connectivity’ option in restore is provided just for this purpose. A restore can be performed on to same platform (different device) with different IP. Hostname will be overwritten. - Can you restore on to completely different platform (5412->CSP) ?
No - Are there any config paramsthat must match?
No. For a restore to start, only PID and NFVIS version should match.
12. Because vm restoration requires registered image, how does “hostaction restore“ recreate this registered image, if there is none during the restoration?
Prior to NFVIS 4.2.1FC3, Registered image is not restored from any backup files (vm backup or host backup), it is restored from original downloaded image.
Starting 4.2.1FC3, in the target system, VNF image registration is created as part of the restore process. VNF image is registration is sourced from .bkup file, VNF is registered and diffs are merged with backing store and VNF deployed.
13. Before vmExport/hostaction_backup, the deployment disk is thin-provisioned_link-clone disk (e.g., small size but needs backingStore file), does restored VM from backup (vmImport or hostaction_restore) will restore this kind of disk?
[Ans] No. The backup (by vmExport or host backup) will merge *backingStore disk* and this small *link-clone disk* into *thin-provision full-clone disk*. This process is not reversible. The restored VM disk is a full-clone disk ( == backingStore + link-clone disk). The only way to restore a link-clone disk is: backup feature is created by copying this disk into backup files, not merging backingStore with link-clone file.
14. Can we change datastore for restored VM?
[Ans] path is hard-coded inside vm backup file, changing datastore is not possible.
15. What disks included in my calculation for end-to-end host restoration?
[Ans] registered_image + backup_file + restore_vm_disk, where:
Backup_file <= gzipped ( registered_image + link_clone disk)
Restore_vm_disk = merged (registered_image + link_clone disk) > registered_image
We tried the host restoration after deleting the registered image, due to the downloaded image is not deleted, so host restore process was successful. If I deleted both download file and registered image, the host restore process will fail. So to me, that does not make any differences on the calculation above.
16. There are four files we concern about for VM restoration:
a. Downloaded package. This can be deleted after registration.
b. Register image. Required by deployment and restore/import process.
c. Backup files.
d. Deployment disk, either disk created from fresh installation or disk restored from backup.
Put vmImport/vmExport aside, if
- Registered image is not existed when doing host restored, disk space need to be bigger than 1+2+3+4 for successful restoration.
- Registered image is existed when doing host restored, disk space need 2+3+4 for successful restoration.
For example, you have one ISRv, two Firewalls, and assume both images takes 1GB physical space, 8 GB max provisioned space. min disk is created right after deployment. Max is time when the whole system running the really long time. on on NFVIS 3.1X and 4.1:
min disk space = 1 x (100k + 1G) + 2 x (100k + 1G) = 3.3 GB
max disk space = 1 x 8G + 2 x 8G = 24 GB
on NFVIS 4.2.1FC3 onwards:
min disk space = 1 x 100k + 2 x 100k + 1GB (ISRv image) + 1GB (Firewall image) = 2.3GB
max disk space = 1 x 8G + 2 x 8G + 1GB (ISRv image) + 1GB (Firewall image) = 26 GB
