cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

UCSD - The "classy way" to delete VMs: Park them in "purgatory" using the power of vDCs!

2314
Views
10
Helpful
12
Comments

Workflow

Description

Shows you how to leverage the power of vDC's in UCSD to offer end-users a way to request VM deletion, but have

the system hold onto their VM's for a period of time before actually automatically removing them.  VMs that are in "purgatory"

can still be accessible, and can still have a unique cost model associated to the business for showback / chargeback.

Prerequisites

Tested on 5.x and 6.x (2 versions attached)

CategoryWorkflow / Configuration
ComponentsUCSD
User InputsVDC

Instructions for Regular Workflow Use:

  1. Download the attached .ZIP file below to your computer. *Remember the location of the saved file on your computer.
  2. Unzip the file on your computer. Should end up with a .WFD file.
  3. Log in to UCS Director as a user that has "system-admin" privileges.
  4. Navigate to "Policies-->Orchestration" and click on "Import".
  5. Click "Browse" and navigate to the location on your computer where the .WFD file resides. Choose the .WFD file and click "Open".
  6. Click "Upload" and then "OK" once the file upload is completed. Then click "Next".
  7. Click the "Select" button next to "Import Workflows". Click the "Check All" button to check all check boxes and then the "Select" button.
  8. Click "Submit".
  9. A new folder should appear in "Policies-->Orchestration" that contains the imported workflow. You will now need to update the included tasks with information about the specific environment.
  10. Follow all configuration steps listed below.

One use case we often get is "Can my end users recover their VMs after they are deleted?   One elegant, policy-driven solution to this is to leverage the power of UCSD's vDCs and policies!

This "VM Purgatory" solution allows an end-user to "Mark" one of their VMs for deletion.   When selected, UCSD will automatically power off their VM, and move it from its current vDC to a "Purgatory vDC" - where it will sit for a configurable amount of time until it is actually automatically deleted.   The end-user will get notified a pre-determined number of times before the VM is actually permanently deleted (could be days, weeks, even months later), and given an opportunity to recover the VM by moving it from the "Purgatory vDC" and back into its original vDC.   Using the "vDC Locking" feature in UCSD, users are not able to actually provision VM's into the vDC by accident.  The purgatory vDC can also have its own cost model if desired, so the cost of the VMs provisioned disk, for example, can still be charged back to the business.  See the instructions below on how to configure this in your own UCSD environment:

First -- make sure that deletion of inactive VM's is actually enabled in your UCSD environment!  Go to "Administration --> System --> Advanced Controls" and scroll all the way to the bottom.  Make sure the following is selected:

1.GIF

Then follow these steps to enable VM Deletion using vDC Policy:

1. Pick an existing vDC in your environment that you want to enable this "VM purgatory" feature on.  In my lab, we will use my "Development vDC".   The first thing you are going to want to do is edit your vDC's existing "end user self service policy" so that users can no longer delete one of their VMs in the traditional sense: Policies --> Virtual/Hypervisor Policies --> Service Delivery --> End User Self Service Policy".  Uncheck "VM Deletion Management"

0.GIF

2. Download and unzip the attached zip file, and go to "policies --> orchestration" and import the vm_purgatory.wfdx file:

1.GIF

3. Next, we need to create two new User VM Action Policies.  Click on the "User VM Action Policy" tab (Policies --> Orchestration). The first policy we will create is for your existing vDC.  If your vDC already has an action policy, edit it, increment the number of actions by one, and add the "Mark VM for Deletion" workflow to it.   If your vDC does not have an existing action policy, click "Add" to create a new policy.   Name it "vDC Action Policy" or something similar and set the number of actions to 1:

4.GIF

Click NEXT and configure the action as below:

5.GIF

Click SUBMIT and then select "ADD" to create a second action policy for your purgatory VDC.  Configure as below:

6.GIF

7.GIF

4. Now we need to create a new "end user self service policy" for our Purgatory vDC to make sure that end-users can't do anything BUT unmark their VMs for deletion.   Click on "Policies --> Virtual/Hypervisor Policies --> Service Requests --> End User Self Service Policy".  Click ADD and name it "Purgatory vDC" or something similar.  Make sure nothing is checked:

8.GIF

5. OPTIONAL STEP: Create a new cost model for the Purgatory vDC.  One of the cool things about this method is that if you are using showback/chargeback, you can create a separate cost model to be applied to VMs that are marked for deletion.  A common function is to continue charging for storage while the VM is waiting to be deleted.  If you'd like to do this, click on "Policies --> Virtual/Hypervisor Policies --> Service Requests --> Cost Model" and select ADD.   Create your cost model for your Purgatory vDC.  I created mine to charge .05 cents per hour per GB of storage while VMs are waiting to be deleted:

9.GIF

6. You need to create a VM Management Policy for your Purgatory vDC.  This is the policy that will govern how long VMs are kept before they are actually permanently deleted, as well as the notification rules for your end users.   Click on "Policies --> Virtual/Hypervisor Policies --> Service Requests --> VM Management Policy" and click ADD to create a new policy.  Make sure you uncheck "Configure VM Lease Notification" and fill out the rest similar to the screenshot below.  I set mine to delete VMs after 7 days, and notify twice with 2 days remaining before deletion:

10.GIF

7. We are now ready to create our Purgatory vDC!  Click on "Policies --> Virtual/Hypervisor Policies --> Virtual Data Centers".  Select the vDC you want to enable this feature for, and select "Clone". In my example, I cloned my Development vDC.   Name the new vDC to "Purgatory vDC", and be sure to click the vDC Locked button!!!  This will not allow end users to actually provision new VMs to this vDC:

12.GIF

Scroll down to the "Policies" section and set the appropriate policies as below.   Click ADD to save your new Purgatory vDC:

11.GIF

8. You need to edit the vDC that you want to enable this feature for (in my example, my Development vDC), and make sure it is pointing to the correct User Action Policy (If your vDC already had a User Action Policy assigned this should be done already. If you created a new User Action Policy for your vDC in step 3 you need to make sure you remember to do this step)!!!   Edit the vDC that you want to enable this feature for, scroll down to policies, and make sure your User Action Policy is set:

13.GIF

9. Almost there!  Last step!  We need to set the "Mark VM For Deletion" workflow to move the selected VM to the newly created Purgatory vDC when it is run!   Go to "Policies --> Orchestration" and open the workflow folder where you imported your workflows in Step 2.   You will probably notice that one of the workflows called "Mark VM for Deletion" has a validation status of "failed":

17.GIF

Click on this workflow and select "Validate".   Click on the edit pencil:

2.GIF

Click "Next" twice to get to the "Task Inputs" page.  Select your Purgatory vDC:

18.GIF

Click submit!

--------

That's it!  You're done!  Now users that can provision to that vDC will have the option to "Mark their VMs for Deletion".  Try it out!

Log in to your end user portal as a service-end user.  The first thing you will notice is that when you try to order a new VM, the "Purgatory vDC" will not show up in the user's drop down because the vDC is locked:

14.GIF

                                                       ^^^^^^^^ Only my development vDC shows up when I try to order a new VM.


When an end-user clicks on a VM that they previously provisioned under "Virtual Resources --> VM", our new "Mark for Deletion" button shows up:

15.GIF

When "Mark VM for Deletion" is submitted for the selected VM, the VM will be shutdown and automatically moved into the "Purgatory vDC", where the VM Management Policy will be inherited.  In my example, if no action is taken I will be notified two days before my VM is actually deleted.  If I take no action, it will be rolled back and permanently deleted.   Here is an example email:

16.GIF

When an end-user looks at his or her VMs in "Virtual Resources --> VMs" they can still see the VMs that have been marked for deletion in the Purgatory vDC.   When they click on one, the only thing that they are able to do is select the "Unmark VM for Deletion" option, which would move the VM from the Purgatory vDC back into a valid vDC, thus preventing it from being wiped out by the VM Management Policy.

19.GIF

Cool stuff!!!

Questions?  Email Jeff Dubois: jefduboi@cisco.com

Comments
Beginner

Hello,

I have a couple of questions about this Purgatory idea as I am implementing it into my environment and try to understand all of the steps it takes behind the scenes (I am still fairly new to UCSD and there seem to be quite a few things going on that I can't trace to any system setting).

1.) After I move VMs into the Purgatory vDC through the "mark for deletion" step I expect them to get deleted 2 days later (1 day deletion for inactivity plus 1 day grace period). However, I have the following scenarios of three VMs scheduled for deletion by three different users:

     - Workflow initiated by system admin results in deleted VM one day after the "mark for deletion" step. (not 2 days, also not exactly 24 hours but 3pm the next day (initiated 11:24am))

     - Workflow initiated by enduser1 results in automated emails at 3pm the day after initiation and every following day at that time asking for enduser1 to approve the deletion which turns out to be the rollback workflow of the VM creation workflow. Notice the VM never gets deleted until I approve it manually, which then the VM gets deleted immediately.

     - Workflow initiated by enduser2 results in VM not getting deleted. I also don't receive emails about that VM needing approval for deletion until enduser1 approves their workflow and has their VM get deleted. Then I receive the same email about approval and can delete the VM after manually accepting the rollback workflow.

How can I automate this if I want to have VMs deleted automatically without approval as well as understand the time of the deletion?

2.) I would like the Purgatory vDC to not be visible to enduser, which I changed. However, the VMs of the endusers that are marked for deletion and consequently are in the Purgatory vDC still are listed under virtual resources for each enduser.

Is it possible to have that no show up so that the VMs seem to be fully deleted from what the enduser can tell and only can be saved by the system admin?

Any help would be great and appreciated! Thanks in advance.

Lukas

Cisco Employee

Hi Lukas,

Are you running UCSD 6.5?

1) In your VM Action Policy, try setting the grace period to 0 and the number of days to delete an inactive VM to 2.  Set the notification period to 1 day.  Let me know if you still see the same inconsistencies after 2 days.

2) Unfortunately not.  With this scenario the VM's will still show up in the list of virtual resources, but because of the end-user policy of the Purgatory vDC, when they click on the VM the only thing they should be able to do is unmark it for deletion.  You could get creative and take that option away, and instead put a button called "request undeletion from admin" or something that send you an email to move the VM back to the original vDC or something.  

If you don't want the end-user to see the VM after they delete, you could try building a workflow that powers off the selected VM and then moves it to a vDC that the end-user does not have access to (a new vDC that belongs to a different group than the end-user is part of).  I've not tried that but it may work.

Beginner

Hi Jeff,

thanks for your quick response! 

Yes, I am running UCSD 6.5 in our environment. I also tried to apply the changes you suggested.

1) The grace period cannot be 0. It says minimum 1 and maximum 7, so I left it at 1 day. I changed the number to delete inactive VM to 2 and the notification period to 1. When I did the same testing procedure as described above, I got a similar response. The first VM scheduled for deletion (9/5 15:44) started the Rollback Provision VM SR just a day later (9/6 18:59). However, it asked again for approval, which I think is because the enduser requested the removal but the system starts that workflow from the admin perspective. The VM initiated by the admin gets deleted without approval needed.

2) I think I tried your last suggestion already. One of my enduser is part of a separate user group that also has an individual vDC assigned to it. So that enduser does not have access to the purgatory vDC but I would still see the powered off VM listed under virtual resources.

Beginner

As an update for my second question in the above comment:

I set up my system as follows to accomplish that the enduser initiating VM deletion does not have any more visibility or manageability of the VM.

To do so you first need two different groups: one for endusers and one for the system admins/techs who manage VMs specifically. The admin group has to be selected now as the owner of the purgatory cDC. The enduser now can't see the purgatory vDC, but would at this stage still see the VM she owns listed under Virtual Resources. As part of that visibility she still can perform the "Unmark VM for Deletion" step. In order to take that option away and have the VM not even show up under Virtual Resources for the enduser I changed the ownership of the VM as part of the "Mark VM for Deletion" workflow. This can be accomplished by a Custom Task listed in the INDEX #85 (Assign VM to Group). Placing that at the bottom of the "Mark VM for Deletion" Workflow and mapping it to the group of the system admins/techs or whoever you would like to have control over the purgatory makes the VM disappear for the enduser.

Cisco Employee

Brilliant!  I need to update the documentation to include that Assign VM To Group task.

So the issue now is just around the grace period?

Cisco Employee

Yes he’s testing again with it…

Yes once you move the VM to a different owner (since it is already in a different VDC – also differend owner) the VM was still visible till it moves owner ship.

Beginner

Yes, in general understanding the timeline from marking a VM for Deletion until getting deleted seems still very random in my environment. In my latest test run they got deleted one day and a couple of hours after moving in to the Purgatory vDC. Orf pointed me to system tasks to understand the couple of hours and how that could vary from time to time.

In my VM Management Policy I had the deletion set to 2 days after VM inactivity and 1 day of grace period, which does not explain the one day though...And yes, I just started another test run with different numbers to get away from "1".

Cisco Employee

We are awaiting the end of incubation period with anticipation

Beginner

The enduser who requested the deletion of the VM got an approval notice this morning. So the times look like:

     - Enduser marks VM for deletion : 9/28/17 10:11 am

     - Enduser gets approval notice : 9/29/17 10:59:06 am

     - Enduser approves Rollback of VM creation : 9/29/17 4:15:33 pm

     - Inactive VM gets deleted out of the purgatory vDC : 9/29/17 4:15:40 pm

I noticed a System Task called PeriodicVMManagementPolicyTask (System Tasks > General) that probably is the issue at hand. This task runs every 4 hours in my environment and typically lasts only one second. However, there was one that got initiated on 9/29/17 10:58:59 am (so right before the approval notice was sent as part of the Rollback workflow execution; and also 24 hours after the first time this Task ran after marking the VM for deletion) and it lasted until 9/29/17 4:15:50 pm, which was just after the VM got approved for deletion and then also was deleted.

Beginner

I installed the patch that came out beginning of this week in hope that would maybe fix some underlying issues with this. However, I ran another test on 10/10 4:29pm and just got the approval notice for the enduser at 4:45 (which is again just after that PeriodicVMManagementPolicyTask ran).

Another detail I thought was interesting based on talking to Orf is that under Virtual Resources when I look at this marked for deletion VM in my purgatory vDC, it does not have a scheduled time for deletion associated with it.

Any further thoughts on this?

Cisco Employee

I don’t have a good answer. I sent this thread to one of our Product TME’s.

Cisco Employee

Hi Lukas,

Engineering has replied that there seem to be several bugs at work here.  They are forwarding the request to engineering to dig into the code, but we need an associated TAC case with the request.

Can I please trouble you to open a TAC case for this issue?

Thank you,

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards
This widget could not be displayed.