NetApp VASA Provider + PowerCLI = Profile Driven Storage with BC\DR requirements.

This year at VMworld Glenn Sizemore and I will be presenting VMworld session BCO3107-SPO – Business Continuity and Disaster Recovery: Choosing the best option for your business. In light of the fact that NetApp will be demonstrating Storage Capability Profiles (SCPs), the NetApp VASA Provider (VP), and VVOLs in the Hands-on Labs, I thought it would be fun to tie-in profile driven storage with managing BC\DR requirements during the VM provisioning process.

What is the VASA Provider?

The basic premise of vSphere APIs for Storage Awareness (VASA) is that the VASA provider (VP), which is created by the storage vendor, surfaces capability information about the storage array. The VP will pass up capabilities like deduplication, replication, snapshot status, RAID level, drive type, and performance (IOps/MBps) capacities. vSphere administrators can then use these profiles to create a VM Storage Profile. VM Storage Profiles are then used to determine which datastores are compatible (support the necessary capabilities) or incompatible (don’t support the required capabilities) when provisioning new VMDKs, performing Storage vMotion operations, cloning a VM, or deploying a VM from a template.

A Use Case Scenario:

The Storage Administrator want to make sure the VI Admin is provisioning VMDKs on datastores with the appropriate capabilities. The VI admin doesn’t want to have to guess the features of every single datastore in the environment.  If the VMDK belongs on the stretched cluster he wants to ensure the DRS Affinity Groups are configured properly to prevent any latency from unnecessarily traversing the inter-cluster links. In the case of DR, he wants to make sure the VM is protected and available for recovery in the event of a disaster.

The Solution:

Using the NetApp VASA provider, vSphere Tags, and a little PowerCLI, we achieve the following:

1. Use NetApp Storage Capability Profiles to map storage features (i.e. deduplication, replication, snapshot status, RAID level, drive type, performance etc) to datastores.
2. Leverage VM Storage Policies configured to combine SCPs and vCenter Tags in order to align storage service tier with BC\DR requirements.
3. Finally, use PowerCLI 5.5 R2 to automate the best practices and other manual configuration:
   • DRS Affinity Group configuration
   • Add newly created VM(s) to the existing SRM Protection Group
   • Initiate an SRM test fail over, including clean-up and sending a status notification.

 

Sample Scripts:

There are three scripts used in the above demo to automate different tasks.  The first one adds newly created virtual machines to the DRS affinity group according to the defined site layout, the second adds virtual machines to the SRM recovery plan protecting a datastore, and the third tests SRM failover with the newly added VMs. In the demo we executed these consecutively, however they do not have to be.

DRS Affinity Group Auto Config

# connection variables
$vcenterIp = 192.168.0.0
$vcenterUser = "administrator@vsphere.local"
$vcenterPassword = "Password!"

# Define the site resource associations
$infrastructure = @{
    # the vCenter cluster to operate on
    'cluster1' = @{
        # the site name
        'site A'= @{
            # hosts belonging to this site
            'hosts' = @('esx1.demo.netapp.com');
            # datastores belonging to this site
            'datastores' = @('metrocluster_siteA_1');
        };
        # repeat as above
        'site B' = @{
            'hosts' = @('esx2.demo.netapp.com');
            'datastores' = @('metrocluster_siteB_1');
        };
    };
    # add additional clusters as needed
}

# import the VMware PowerCli module
if (!(Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) { 
    Add-PSSnapin vmware.vimautomation.core
}

# eliminate invalid certificate errors, only connect to one vCenter at a time
Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore  -Scope session -Confirm:$false | Out-Null

# connect to vCenter
Connect-VIServer -Server $vcenterIp -User $vcenterUser -Password $vcenterPassword | Out-Null
Write-Host "Connected to vCenter server $($vcenterIp)"

while ($true) {
    # loop through our clusters
    $infrastructure.GetEnumerator() | %{ 
        # get the cluster object
        $clusterName = $_.Key
        
        Write-Host "Getting cluster $($clusterName) data..."
        $cluster = Get-Cluster $clusterName
        
        # create the specification objects for making the modification
        $clusterSpec = New-Object VMware.Vim.ClusterConfigSpecEx
        
        # for each of the sites, add the hosts and VMs to the spec
        # value is the hash with keys equal to each site name
        $_.Value.GetEnumerator() | %{
            $siteName = $_.Key
            
            # value is the hash with keys of "hosts" and "datastores"
            $hostGroupSpec = New-Object VMware.Vim.ClusterGroupSpec
            $hostGroupSpec.operation = "edit"
            $hostGroupSpec.Info = New-Object VMware.Vim.ClusterHostGroup
            $hostGroupSpec.Info.Name = "$($siteName) Hosts"
            Write-Host "  Editing host group $($hostGroupSpec.Info.Name)"
            
            # make sure all of the hosts have been added
            $_.Value.hosts | %{
                # add each host MoRef to the list
                $hostGroupSpec.Info.Host += (Get-VMHost -Name $_).ExtensionData.MoRef
                Write-Host "    Adding host $($_) to DRS group"
            }
            
            $clusterSpec.GroupSpec += $hostGroupSpec

            $vmGroupSpec = New-Object VMware.Vim.ClusterGroupSpec
            $vmGroupSpec.operation = "edit"
            $vmGroupSpec.Info = New-Object VMware.Vim.ClusterVmGroup
            $vmGroupSpec.Info.Name = "$($siteName) VMs"
            Write-Host "  Editing VM group $($vmGroupSpec.Info.Name)"
            
            # make sure all the VMs in the datastores are added
            $_.Value.datastores | %{
                Write-Host "    Adding VMs from datastore $($_)"
                
                # get the VMs in the datastore
                Get-Datastore -Name $_ | Get-VM | %{ 
                    # add them to the list
                    $vmGroupSpec.Info.VM += $_.Extensiondata.MoRef
                    Write-Host "      Adding VM $($_.Name)"
                }
            }
            
            $clusterSpec.GroupSpec += $vmGroupSpec
            
            # now we want to associate the site VMs with the site hosts
            $ruleSpec = New-Object VMware.Vim.ClusterRuleSpec
            $ruleSpec.operation = "edit"
            $ruleSpec.Info = New-Object VMware.Vim.ClusterVmHostRuleInfo
            $ruleSpec.Info.enabled = $true
            $ruleSpec.Info.name = "$($siteName) Affinity Rules"
            $ruleSpec.Info.mandatory = $false
            $ruleSpec.Info.vmGroupName = "$($siteName) VMs"
            $ruleSpec.Info.affineHostGroupName = "$($siteName) Hosts"
            
            Write-Host "  Associating host and VM groups..."
            
            $clusterSpec.RulesSpec += $ruleSpec
        }
        
        Write-Host "  Implementing new configuration..."
        
        # implement the rules
        $cluster.ExtensionData.ReconfigureComputeResource( $clusterSpec, $true )
        
    }

    Write-Host "Iteration complete!"
    
    Write-Host "Sleeping..."
    Start-Sleep -Seconds 60
}

Add Virtual Machines to SRM Failover Group

# connection variables
$vcenter = "192.168.0."
$user = "administrator@vsphere.local"
$pass = "Password!"

# we can only update local plans...not remote
$plan = "NYC"

###############

$credential = New-Object System.Management.Automation.PSCredential ($user, (ConvertTo-SecureString $pass -AsPlainText -Force))

if (! (Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) {
    Add-PSSnapin vmware.vimautomation.core
}

# eliminate invalid certificate errors, only connect to one vCenter at a time
Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore  -Scope session -Confirm:$false | Out-Null

# Connect to vCenter
Connect-VIServer -Server $vcenter -Credential $credential | Out-Null

# Connct to SRM
$srm = Connect-SrmServer -RemoteCredential $credential -Credential $credential
$api = $srm.ExtensionData

# Get the desired datastores...this is to compare later
$datastores = @{}

# just SRM
Get-Datastore -Tag "SRM" | %{ 
    $datastores.Add($_.Name, @{})
}

# for each protection plan, get the datatores and VMs
$protectionGroups = $api.Protection.ListProtectionGroups();

$protectionGroups | %{
    $plan = $_
    
    Write-Host "Checking protected entities for plan $($plan.GetInfo().Name)"
    
    Write-Host "  Getting protected datastores"
    $planDatastores = $plan.ListProtectedDatastores()
    
    Write-Host "    Found $($planDatastores.count) datastores"
    
    $planDatastores | %{
        $ds = $_
        try {
            $ds.UpdateViewData()
        } catch {
            # a remote datastore will fail, we're just going to skip the failure
            return
        }
        
        $dsName = $ds.Name
        
        Write-Host "    Found protected datastore $($dsName)"
        
        # skip un-tagged datastores, even if they're protected
        if (!($datastores.keys -contains $dsName)) {
            Write-Host -ForegroundColor red "      Skipping datastore - it's not tagged!"
            return
        }
        
        $datastores.$dsName.Add("protectionGroup", $plan)
        $datastores.$dsName.Add("object", (Get-VIObjectByVIView $ds))
        
        # the datastore view has references to the VMs, let's collect all
        # vms in the datastore now
        $dsVms = @()
        $ds.Vm | %{
            $dsVms += Get-VIObjectByVIView $_
        }
        
        # add the array of VMs to the hash
        $datastores.$dsName.Add("currentVms", $dsVms)
        
        # a placeholder for later
        $datastores.$dsName.Add("protectedVms", @())
    }
    
    # get the protected VMs and associate them with the datastore
    Write-Host "  Getting protected Virtual Machines"
    $plan.ListProtectedVms() | %{
        try {
            $_.Vm.UpdateViewData()
        } catch {
            # skip remote VMs, which will fail
            return
        }
        
        $vm = Get-VIObjectByVIView $_.Vm
        
        # ignore templates
        if ($vm.GetType().name -eq "TemplateImpl") {
            return
        }
        
        Write-Host "    Found protected VM $($vm.Name)"
        
        # get the datastore
        $vmDatastore = $vm | Get-Datastore
        $vmDatastoreName = $vmDatastore.Name
        
        Write-Host "      VM uses datastore $($vmDatastoreName)"
        
        #skip un-tagged datastores, even if they're protected
        if (!($datastores.keys -contains $vmDatastoreName)) {
            Write-Host -ForegroundColor red "      Skipping VM - datastore not tagged!"
            return
        }
        
        $datastores.$vmDatastoreName.protectedVms += $vm
    }
}

Write-Host "Data collection complete..."

#
# at this point our hash named "datastores" contains keys equal to the names
# of each of the datastores.  Each of those keys has a value that is another
# hash, which has 4 keys: object (the datastore object itself), protectionGroup
# (the protection group which is protecting the VMs in the datastore), 
# protectedVms (an array of all the protected VMs), and currentVms (an array of 
# all the VMs in the datastore currently)
#

# Compare the VMs in the datastore with the VMs in the SRM plan
$datastores.GetEnumerator() | %{ 
    $datastoreName = $_.Key
    $datastoreData = $_.Value
    
    Write-Host "Checking datastore $($datastoreName) for unprotected VMs"
    
    # an empty array to hold all the VMs to be protected
    $vmsToProtect = @()
    
    # loop through each of the VMs in the datastore, check to see if it's being
    # protected
    $datastoreData.currentVms | %{
        if (! $datastoreData.protectedVms -contains $_) {
            Write-Host -ForegroundColor yellow "  VM $($_.Name) is not protected"
            
            $spec = New-Object VMware.VimAutomation.Srm.Views.SrmProtectionGroupVmProtectionSpec
            $spec.Vm = $_.ExtensionData.MoRef
            $vmsToProtect += $spec
        }
    }
    
    if ($vmsToProtect.count -gt 0) {
        # add any missing VMs to the SRM plan
        Write-Host -ForegroundColor green "  Protecting VMs in datastore $($datastoreName) using plan $($datastoreData.protectionGroup.GetInfo().Name)"
        $task = $datastoreData.protectionGroup.ProtectVms( $vmsToProtect )
        while (-not $task.IsComplete()) { 
            Write-Host "  Waiting for task to complete..."
            sleep -Seconds 5
        }
    } else {
        Write-Host "  No unprotected VMs found"
    }
}

# disconnect
Disconnect-SrmServer -Confirm:$false
Disconnect-VIServer -Confirm:$false

Start an SRM Test and Cleanup

# connection variables
$vcenter = "192.168.0.0"
$user = "administrator@vsphere.local"
$pass = "Password!"

# the name of the recovery plan to test
$RecoveryPlanName = "NYC"

# how many minutes to wait for the test and cleanup to execute
$waitTime = 10

# email settings
$sendEmail = $true
$emailFrom = "srm_test@netapp.com"
$emailTo = @("you@yourcompany.com")
$emailCc = @("minion1@yourcompany.com","hmfic@yourcompany.com")
$emailSubject = "SRM Test Report $(Get-Date -Format yyyy-MM-dd)"
$emailSmtpServer = "smtp.yourcompany.com"
$emailSmtpServerPort = 25
$emailSmtpUser = 'service_account'
$emailSmtpPassword = 'Password!'
$emailSmtpSsl = $true

# ######### End of Editable Section ######### #

function Send-Report( $message ) {
    # create the email
    $email = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo)
    $email.IsBodyHtml = $True
    $email.body = @"

SRM Test Report

$message

          
        
"@

    # add any CC recipients
    if ($emailCc.count -gt 0) {
    $emailCc | %{ $email.CC.Add($_) }
    }

    # craft the rest of the message
    $email.subject = $EmailSubject
    
    # send the message
    $smtp = New-Object System.Net.Mail.SmtpClient($emailSmtpServer,$emailSmtpServerPort)
    $smtp.EnableSsl = $emailSmtpSsl
    $smtp.credentials = New-Object System.Net.NetworkCredential( $emailSmtpUser,$emailSmtpPassword) 
    $smtp.Send($email)
}

$credential = New-Object System.Management.Automation.PSCredential ($user, (ConvertTo-SecureString $pass -AsPlainText -Force))

if (! (Get-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue)) {
    Add-PSSnapin vmware.vimautomation.core
}

# eliminate invalid certificate errors, only connect to one vCenter at a time
Set-PowerCLIConfiguration -DefaultVIServerMode Single -InvalidCertificateAction Ignore  -Scope session -Confirm:$false | Out-Null

# Connect to vCenter
Connect-VIServer -Server $vcenter -Credential $credential | Out-Null

# connect to srm and get the api moref
$srm = Connect-SrmServer -Credential $credential -RemoteCredential $credential -SrmServerAddress 192.168.0.32
$srmapi = $srm.ExtensionData

# find the recovery plan we want
$recoveryplan = $srmapi.Recovery.ListPlans() | ?{ $_.GetInfo().Name -eq $RecoveryPlanName }
Write-Host "Found plan: $($recoveryplan.Name)"

# if it's in the ready state, do a test
if ($recoveryplan.getInfo().State -eq "Ready") {
    Write-Host "Plan is in the ready state, continuing"
    
    # set the mode to test (1 = test)
    $rpmode = New-Object VMware.VimAutomation.Srm.Views.SrmRecoveryPlanRecoveryMode
    $rpmode.value__ = 1
    
    # trigger the test
    Write-Host "Starting test..." -NoNewline
    $recoveryplan.Start( $rpmode )
    
    # wait until it's done or $waittime minutes have passed
    $start = get-date
    while ((New-TimeSpan $start).TotalMinutes -le $waittime) {
        if ($recoveryplan.GetInfo().State -ne "running") {
            break
        }
        
        Write-Host "." -NoNewline
        sleep -seconds 5
    }
    Write-Host "DONE!"

    # when the test is done the state *should* be "NeedsCleanup", if not, an error happened
    $history = $srmapi.Recovery.GetHistory( $recoveryplan.MoRef )
    $last = $history.GetRecoveryResult(1)[0]

    $testresult = $last.ResultState
    Write-Host "Last result was: $($testresult)"
    
    if ($last.ResultState -ne "Success") {
        # do something to figure out what went wrong
        $mailmessage = "something went wrong during the test!"
        Write-Host $mailmessage
        
        if ($sendEmail) {
            Send-Report $mailmessage
        }
        
    } else {
        # get the report, if desired
        #[XML]$resultXML = $history.RetrieveStatus( $last.Key, 0, $history.GetResultLength( $last.Key ) )
        
        # trigger cleanup
        $rpmode.value__ = 2
        Write-Host "Starting cleanup..." -NoNewline
        $recoveryplan.Start($rpmode);
        
        # wait for cleaup to finish/fail
        $start = Get-Date
        while ((New-TimeSpan $start).TotalMinutes -le $waittime) {
            if ($recoveryplan.GetInfo().State -ne "running") {
                break
            }
        
            Write-Host "." -NoNewline
            sleep -seconds 5
        }
        write-host "DONE!"
                
        # get cleanup result
        $cleanupresult = $history.GetRecoveryResult(1)[0].ResultState
        Write-Host "Cleanup result: $($cleanupresult)"
        
        # get the report, if desired
        #[XML]$cleanupResultXML = $history.RetrieveStatus( $history.GetRecoveryResult(1)[0].Key, 0, $history.GetResultLength( $history.GetRecoveryResult(1)[0].Key ) )
                
        # craft the message
        $mailmessage = "The result of the test was $($testresult), the result of cleanup was $($cleanupresult)."
        
        # send the message
        if ($sendEmail) {
            Send-Report $mailmessage
        } else {
            Write-Host $mailmessage
        }
    }
} else {
    # send email saying some kind of error because recovery plan not ready
    $mailmessage = "Unable to proceed, plan in state " + $recoveryplan.getInfo().State
    Write-Host $mailmessage
    
    if ($sendEmail) {
        Send-Report $mailmessage 
    }
}

Conclusion

PowerShell is a powerful automator and can elminiate many of the tedious burdens placed on VI admins who are trying to ensure that business applications meet the BC/DR SLAs that have been placed on them. Automation ensures that these rules are adhered to, kept in place, and done in a consistent manner, regardless of who (or what) is deploying virtual machines in your environment.

If you are going to VMworld 2014 be sure to swing by NetApp Booth 1205 and check out the NetApp VASA Provider in the Hands-On Labs. We have experts on automation, disaster recovery, business continuity, and any other subject you want to talk about. Be sure to stop by to schedule your time with any of the Technical Marketing team from NetApp supporting VMware integration!