This year at VMworld Glenn Sizemore and I will be presenting VMworld session BCO3107-SPO – Business Continuity and Disaster Recovery: Choosing the best option for your business. In light of the fact that NetApp will be demonstrating Storage Capability Profiles (SCPs), the NetApp VASA Provider (VP), and VVOLs in the Hands-on Labs, I thought it would be fun to tie-in profile driven storage with managing BC\DR requirements during the VM provisioning process.
What is the VASA Provider?
The basic premise of vSphere APIs for Storage Awareness (VASA) is that the VASA provider (VP), which is created by the storage vendor, surfaces capability information about the storage array. The VP will pass up capabilities like deduplication, replication, snapshot status, RAID level, drive type, and performance (IOps/MBps) capacities. vSphere administrators can then use these profiles to create a VM Storage Profile. VM Storage Profiles are then used to determine which datastores are compatible (support the necessary capabilities) or incompatible (don’t support the required capabilities) when provisioning new VMDKs, performing Storage vMotion operations, cloning a VM, or deploying a VM from a template.
A Use Case Scenario:
The Storage Administrator want to make sure the VI Admin is provisioning VMDKs on datastores with the appropriate capabilities. The VI admin doesn’t want to have to guess the features of every single datastore in the environment. If the VMDK belongs on the stretched cluster he wants to ensure the DRS Affinity Groups are configured properly to prevent any latency from unnecessarily traversing the inter-cluster links. In the case of DR, he wants to make sure the VM is protected and available for recovery in the event of a disaster.
The Solution:
Using the NetApp VASA provider, vSphere Tags, and a little PowerCLI, we achieve the following:
- Use NetApp Storage Capability Profiles to map storage features (i.e. deduplication, replication, snapshot status, RAID level, drive type, performance etc) to datastores.
- Leverage VM Storage Policies configured to combine SCPs and vCenter Tags in order to align storage service tier with BC\DR requirements.
- Finally, use PowerCLI 5.5 R2 to automate the best practices and other manual configuration:
- DRS Affinity Group configuration
- Add newly created VM(s) to the existing SRM Protection Group
- Initiate an SRM test fail over, including clean-up and sending a status notification.
Sample Scripts:
There are three scripts used in the above demo to automate different tasks. The first one adds newly created virtual machines to the DRS affinity group according to the defined site layout, the second adds virtual machines to the SRM recovery plan protecting a datastore, and the third tests SRM failover with the newly added VMs. In the demo we executed these consecutively, however they do not have to be.
DRS Affinity Group Auto Config
# connection variables $vcenterIp = 192.168.0.0 $vcenterUser = "administrator@vsphere.local" $vcenterPassword = "Password!" # Define the site resource associations $infrastructure = @{ # the vCenter cluster to operate on 'cluster1' = @{ # the site name 'site A'= @{ # hosts belonging to this site 'hosts' = @('esx1.demo.netapp.com'); # datastores belonging to this site 'datastores' = @('metrocluster_siteA_1'); }; # repeat as above 'site B' = @{ 'hosts' = @('esx2.demo.netapp.com'); 'datastores' = @('metrocluster_siteB_1'); }; }; # add additional clusters as needed } # import the VMware PowerCli module if (!(Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) { Add-PSSnapin vmware.vimautomation.core } # eliminate invalid certificate errors, only connect to one vCenter at a time Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore -Scope session -Confirm:$false | Out-Null # connect to vCenter Connect-VIServer -Server $vcenterIp -User $vcenterUser -Password $vcenterPassword | Out-Null Write-Host "Connected to vCenter server $($vcenterIp)" while ($true) { # loop through our clusters $infrastructure.GetEnumerator() | %{ # get the cluster object $clusterName = $_.Key Write-Host "Getting cluster $($clusterName) data..." $cluster = Get-Cluster $clusterName # create the specification objects for making the modification $clusterSpec = New-Object VMware.Vim.ClusterConfigSpecEx # for each of the sites, add the hosts and VMs to the spec # value is the hash with keys equal to each site name $_.Value.GetEnumerator() | %{ $siteName = $_.Key # value is the hash with keys of "hosts" and "datastores" $hostGroupSpec = New-Object VMware.Vim.ClusterGroupSpec $hostGroupSpec.operation = "edit" $hostGroupSpec.Info = New-Object VMware.Vim.ClusterHostGroup $hostGroupSpec.Info.Name = "$($siteName) Hosts" Write-Host " Editing host group $($hostGroupSpec.Info.Name)" # make sure all of the hosts have been added $_.Value.hosts | %{ # add each host MoRef to the list $hostGroupSpec.Info.Host += (Get-VMHost -Name $_).ExtensionData.MoRef Write-Host " Adding host $($_) to DRS group" } $clusterSpec.GroupSpec += $hostGroupSpec $vmGroupSpec = New-Object VMware.Vim.ClusterGroupSpec $vmGroupSpec.operation = "edit" $vmGroupSpec.Info = New-Object VMware.Vim.ClusterVmGroup $vmGroupSpec.Info.Name = "$($siteName) VMs" Write-Host " Editing VM group $($vmGroupSpec.Info.Name)" # make sure all the VMs in the datastores are added $_.Value.datastores | %{ Write-Host " Adding VMs from datastore $($_)" # get the VMs in the datastore Get-Datastore -Name $_ | Get-VM | %{ # add them to the list $vmGroupSpec.Info.VM += $_.Extensiondata.MoRef Write-Host " Adding VM $($_.Name)" } } $clusterSpec.GroupSpec += $vmGroupSpec # now we want to associate the site VMs with the site hosts $ruleSpec = New-Object VMware.Vim.ClusterRuleSpec $ruleSpec.operation = "edit" $ruleSpec.Info = New-Object VMware.Vim.ClusterVmHostRuleInfo $ruleSpec.Info.enabled = $true $ruleSpec.Info.name = "$($siteName) Affinity Rules" $ruleSpec.Info.mandatory = $false $ruleSpec.Info.vmGroupName = "$($siteName) VMs" $ruleSpec.Info.affineHostGroupName = "$($siteName) Hosts" Write-Host " Associating host and VM groups..." $clusterSpec.RulesSpec += $ruleSpec } Write-Host " Implementing new configuration..." # implement the rules $cluster.ExtensionData.ReconfigureComputeResource( $clusterSpec, $true ) } Write-Host "Iteration complete!" Write-Host "Sleeping..." Start-Sleep -Seconds 60 }
Add Virtual Machines to SRM Failover Group
# connection variables $vcenter = "192.168.0." $user = "administrator@vsphere.local" $pass = "Password!" # we can only update local plans...not remote $plan = "NYC" ############### $credential = New-Object System.Management.Automation.PSCredential ($user, (ConvertTo-SecureString $pass -AsPlainText -Force)) if (! (Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) { Add-PSSnapin vmware.vimautomation.core } # eliminate invalid certificate errors, only connect to one vCenter at a time Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore -Scope session -Confirm:$false | Out-Null # Connect to vCenter Connect-VIServer -Server $vcenter -Credential $credential | Out-Null # Connct to SRM $srm = Connect-SrmServer -RemoteCredential $credential -Credential $credential $api = $srm.ExtensionData # Get the desired datastores...this is to compare later $datastores = @{} # just SRM Get-Datastore -Tag "SRM" | %{ $datastores.Add($_.Name, @{}) } # for each protection plan, get the datatores and VMs $protectionGroups = $api.Protection.ListProtectionGroups(); $protectionGroups | %{ $plan = $_ Write-Host "Checking protected entities for plan $($plan.GetInfo().Name)" Write-Host " Getting protected datastores" $planDatastores = $plan.ListProtectedDatastores() Write-Host " Found $($planDatastores.count) datastores" $planDatastores | %{ $ds = $_ try { $ds.UpdateViewData() } catch { # a remote datastore will fail, we're just going to skip the failure return } $dsName = $ds.Name Write-Host " Found protected datastore $($dsName)" # skip un-tagged datastores, even if they're protected if (!($datastores.keys -contains $dsName)) { Write-Host -ForegroundColor red " Skipping datastore - it's not tagged!" return } $datastores.$dsName.Add("protectionGroup", $plan) $datastores.$dsName.Add("object", (Get-VIObjectByVIView $ds)) # the datastore view has references to the VMs, let's collect all # vms in the datastore now $dsVms = @() $ds.Vm | %{ $dsVms += Get-VIObjectByVIView $_ } # add the array of VMs to the hash $datastores.$dsName.Add("currentVms", $dsVms) # a placeholder for later $datastores.$dsName.Add("protectedVms", @()) } # get the protected VMs and associate them with the datastore Write-Host " Getting protected Virtual Machines" $plan.ListProtectedVms() | %{ try { $_.Vm.UpdateViewData() } catch { # skip remote VMs, which will fail return } $vm = Get-VIObjectByVIView $_.Vm # ignore templates if ($vm.GetType().name -eq "TemplateImpl") { return } Write-Host " Found protected VM $($vm.Name)" # get the datastore $vmDatastore = $vm | Get-Datastore $vmDatastoreName = $vmDatastore.Name Write-Host " VM uses datastore $($vmDatastoreName)" #skip un-tagged datastores, even if they're protected if (!($datastores.keys -contains $vmDatastoreName)) { Write-Host -ForegroundColor red " Skipping VM - datastore not tagged!" return } $datastores.$vmDatastoreName.protectedVms += $vm } } Write-Host "Data collection complete..." # # at this point our hash named "datastores" contains keys equal to the names # of each of the datastores. Each of those keys has a value that is another # hash, which has 4 keys: object (the datastore object itself), protectionGroup # (the protection group which is protecting the VMs in the datastore), # protectedVms (an array of all the protected VMs), and currentVms (an array of # all the VMs in the datastore currently) # # Compare the VMs in the datastore with the VMs in the SRM plan $datastores.GetEnumerator() | %{ $datastoreName = $_.Key $datastoreData = $_.Value Write-Host "Checking datastore $($datastoreName) for unprotected VMs" # an empty array to hold all the VMs to be protected $vmsToProtect = @() # loop through each of the VMs in the datastore, check to see if it's being # protected $datastoreData.currentVms | %{ if (! $datastoreData.protectedVms -contains $_) { Write-Host -ForegroundColor yellow " VM $($_.Name) is not protected" $spec = New-Object VMware.VimAutomation.Srm.Views.SrmProtectionGroupVmProtectionSpec $spec.Vm = $_.ExtensionData.MoRef $vmsToProtect += $spec } } if ($vmsToProtect.count -gt 0) { # add any missing VMs to the SRM plan Write-Host -ForegroundColor green " Protecting VMs in datastore $($datastoreName) using plan $($datastoreData.protectionGroup.GetInfo().Name)" $task = $datastoreData.protectionGroup.ProtectVms( $vmsToProtect ) while (-not $task.IsComplete()) { Write-Host " Waiting for task to complete..." sleep -Seconds 5 } } else { Write-Host " No unprotected VMs found" } } # disconnect Disconnect-SrmServer -Confirm:$false Disconnect-VIServer -Confirm:$false
Start an SRM Test and Cleanup
# connection variables $vcenter = "192.168.0.0" $user = "administrator@vsphere.local" $pass = "Password!" # the name of the recovery plan to test $RecoveryPlanName = "NYC" # how many minutes to wait for the test and cleanup to execute $waitTime = 10 # email settings $sendEmail = $true $emailFrom = "srm_test@netapp.com" $emailTo = @("you@yourcompany.com") $emailCc = @("minion1@yourcompany.com","hmfic@yourcompany.com") $emailSubject = "SRM Test Report $(Get-Date -Format yyyy-MM-dd)" $emailSmtpServer = "smtp.yourcompany.com" $emailSmtpServerPort = 25 $emailSmtpUser = 'service_account' $emailSmtpPassword = 'Password!' $emailSmtpSsl = $true # ######### End of Editable Section ######### # function Send-Report( $message ) { # create the email $email = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo) $email.IsBodyHtml = $True $email.body = @"
SRM Test Report
$message
"@ # add any CC recipients if ($emailCc.count -gt 0) { $emailCc | %{ $email.CC.Add($_) } } # craft the rest of the message $email.subject = $EmailSubject # send the message $smtp = New-Object System.Net.Mail.SmtpClient($emailSmtpServer,$emailSmtpServerPort) $smtp.EnableSsl = $emailSmtpSsl $smtp.credentials = New-Object System.Net.NetworkCredential( $emailSmtpUser,$emailSmtpPassword) $smtp.Send($email) } $credential = New-Object System.Management.Automation.PSCredential ($user, (ConvertTo-SecureString $pass -AsPlainText -Force)) if (! (Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) { Add-PSSnapin vmware.vimautomation.core } # eliminate invalid certificate errors, only connect to one vCenter at a time Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore -Scope session -Confirm:$false | Out-Null # Connect to vCenter Connect-VIServer -Server $vcenter -Credential $credential | Out-Null # connect to srm and get the api moref $srm = Connect-SrmServer -Credential $credential -RemoteCredential $credential -SrmServerAddress 192.168.0.32 $srmapi = $srm.ExtensionData # find the recovery plan we want $recoveryplan = $srmapi.Recovery.ListPlans() | ?{ $_.GetInfo().Name -eq $RecoveryPlanName } Write-Host "Found plan: $($recoveryplan.Name)" # if it's in the ready state, do a test if ($recoveryplan.getInfo().State -eq "Ready") { Write-Host "Plan is in the ready state, continuing" # set the mode to test (1 = test) $rpmode = New-Object VMware.VimAutomation.Srm.Views.SrmRecoveryPlanRecoveryMode $rpmode.value__ = 1 # trigger the test Write-Host "Starting test..." -NoNewline $recoveryplan.Start( $rpmode ) # wait until it's done or $waittime minutes have passed $start = get-date while ((New-TimeSpan $start).TotalMinutes -le $waittime) { if ($recoveryplan.GetInfo().State -ne "running") { break } Write-Host "." -NoNewline sleep -seconds 5 } Write-Host "DONE!" # when the test is done the state *should* be "NeedsCleanup", if not, an error happened $history = $srmapi.Recovery.GetHistory( $recoveryplan.MoRef ) $last = $history.GetRecoveryResult(1)[0] $testresult = $last.ResultState Write-Host "Last result was: $($testresult)" if ($last.ResultState -ne "Success") { # do something to figure out what went wrong $mailmessage = "something went wrong during the test!" Write-Host $mailmessage if ($sendEmail) { Send-Report $mailmessage } } else { # get the report, if desired #[XML]$resultXML = $history.RetrieveStatus( $last.Key, 0, $history.GetResultLength( $last.Key ) ) # trigger cleanup $rpmode.value__ = 2 Write-Host "Starting cleanup..." -NoNewline $recoveryplan.Start($rpmode); # wait for cleaup to finish/fail $start = Get-Date while ((New-TimeSpan $start).TotalMinutes -le $waittime) { if ($recoveryplan.GetInfo().State -ne "running") { break } Write-Host "." -NoNewline sleep -seconds 5 } write-host "DONE!" # get cleanup result $cleanupresult = $history.GetRecoveryResult(1)[0].ResultState Write-Host "Cleanup result: $($cleanupresult)" # get the report, if desired #[XML]$cleanupResultXML = $history.RetrieveStatus( $history.GetRecoveryResult(1)[0].Key, 0, $history.GetResultLength( $history.GetRecoveryResult(1)[0].Key ) ) # craft the message $mailmessage = "The result of the test was $($testresult), the result of cleanup was $($cleanupresult)." # send the message if ($sendEmail) { Send-Report $mailmessage } else { Write-Host $mailmessage } } } else { # send email saying some kind of error because recovery plan not ready $mailmessage = "Unable to proceed, plan in state " + $recoveryplan.getInfo().State Write-Host $mailmessage if ($sendEmail) { Send-Report $mailmessage } }
Conclusion
PowerShell is a powerful automator and can elminiate many of the tedious burdens placed on VI admins who are trying to ensure that business applications meet the BC/DR SLAs that have been placed on them. Automation ensures that these rules are adhered to, kept in place, and done in a consistent manner, regardless of who (or what) is deploying virtual machines in your environment.
If you are going to VMworld 2014 be sure to swing by NetApp Booth 1205 and check out the NetApp VASA Provider in the Hands-On Labs. We have experts on automation, disaster recovery, business continuity, and any other subject you want to talk about. Be sure to stop by to schedule your time with any of the Technical Marketing team from NetApp supporting VMware integration!