Create AAA Hybrid Worker Group On VMSS With PowerShell - Part 1

Part 2, Part 3, Part4

What is AAA?

Azure Automation Account

Why Hybrid Work Group?

Runbooks in Azure Automation can run on either an Azure sandbox or a Hybrid Runbook Worker. The sandbox is shared by Automation accounts and resource limitations with an Azure sandbox make it a big joke when running some serious workloads in the runbooks. There are some other known limitations with Azure sandbox, for example, any cmdlets that perform full stack network operations are not supported in the sandbox.

However, the more practical reason of using Hybrid Runbook Worker instead of the default Azure sandbox, except the performance scalability, is it's the only option if your runbooks need to access resources in your on-premises environment or in your Azure Virtual network subnets, for example, your SQL Managed Instances.

Why VMSS?

You can certainly deploy a Hybrid Runbook Worker on a regular Windows or Linux machine (links provided below), but what makes this article unique and outstanding is that it shows the way to install and configure a Azure Automation Hybrid Runbook Worker group using an Azure Virtual Machine Scale Set (VMSS) which brings High Availability, Load Balancing and Autoscale to the worker group. Using VMSS instead of standalone VMs makes so much sense in terms of dynamic performance management and cost saving.

Solution Diagram:

Ref:

Deploy an agent-based Windows Hybrid Runbook Worker in Automation

Deploy an agent-based Linux Hybrid Runbook Worker in Automation

Deploy an extension-based Windows or Linux User Hybrid Runbook Worker in Automation (Preview)

GitHub Repo (currently private, available upon request):

https://github.com/glindba/Hybrid-Runbook-Worker-With-VMSS.git

Step 1: Let's get things organized in Azure using a resource group with tags

$SubscriptionId = '12345678-c684-40f6-a4d7-0123456789ab'
Set-AzContext -SubscriptionId $SubscriptionId

# Create a resource group if it doesn't exist

$RGName = "MyTest-CUS-RG01"

$Location = "centralus"

$Tags = @{
    CostCenter = 1688;
    Purpose    = 'GLinDBA DPA Test Environment';
    Owner      = 'GLinDBA_DPA'
}

if (!(Get-AzResourceGroup -Name $RGName -ErrorAction SilentlyContinue)) {
    New-AzResourceGroup -Name $RGName -Location $Location -Tag $Tags
}

Step 2: Create an Azure Automation Account

# Create an Azure automation account if it doesn't exist

$AutoAcctName = "MyTest-CUS-AutoAcct01"

if (!(Get-AzAutomationAccount -ResourceGroupName $RGName -Name $AutoAcctName -ErrorAction SilentlyContinue)) {
    $MyTestAutoAcct = New-AzAutomationAccount -ResourceGroupName $RGName -Name $AutoAcctName -Location $Location
}

Step 3: Add Az modules into the automation account from the PowerShell Gallery

Importing a module may take several minutes. So only add the Az modules that are needed by the cmdlets in your runbooks instead of the entire Az.Automation module. Because Az.Accounts is a dependency for the other Az modules, be sure to import this module before any others.

$ModuleInfo = Get-InstalledModule -Name Az.Accounts | Select-Object Name, Version

if (!(Get-AzAutomationModule -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $ModuleInfo.Name -ErrorAction SilentlyContinue )) {
    New-AzAutomationModule -AutomationAccountName $AutoAcctName `
        -ResourceGroupName $RGName `
        -Name $ModuleInfo.Name `
        -ContentLinkUri "https://www.powershellgallery.com/api/v2/package/$($ModuleInfo.Name)/$($ModuleInfo.Version)"
}

$ImportModules = @(
    'Az.Automation',
    'Az.Compute',
    'Az.KeyVault',
    'Az.Network',
    'Az.Resources',
    'Az.Sql',
    'Az.Storage')

Get-InstalledModule | Where-Object Name -in $ImportModules | ForEach-Object {
    $moduleName = $_.Name
    $moduleVersion = $_.Version
    if (!(Get-AzAutomationModule -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $moduleName -ErrorAction SilentlyContinue )) {
        New-AzAutomationModule -AutomationAccountName $AutoAcctName `
            -ResourceGroupName $RGName `
            -Name $moduleName `
            -ContentLinkUri "https://www.powershellgallery.com/api/v2/package/$moduleName/$moduleVersion"
    }
}

Import update: the new Automation accounts now have the latest version of the PowerShell Az module imported by default which includes the existing 24 AzureRM modules and 60+ Az modules.

Step 4: Create or update a Run As Account

The tasks in this step need some specific privileges and permissions, for example, an Application administrator in Azure Active Directory and an Owner in a subscription. To configure a Run As Account, the script deploys the following resources in sequence:

1. Create a new self-assigned certificate and store it in "Cert:\LocalMachine\My"

2. Export a PFX certificate under My store from the local machine account into a file with password

3. Generate a new base64 encoded public X509 certificate using the PFX certificate created in the preceding step

4. Create a new Azure application

5. Add a new credential to the Azure application using the X509 certificate generated in the previous step

6. Create a new Azure AD service principal and associate it with the application created in the previous step.

7. The service principal should have already been automatically assigned the Contributor role on the current subscription. If that isn't the case, the script will try to assign the Contributor role to the principal

8. Create a certificate asset in the Azure Automation account

9. Populate the ConnectionFieldValues hash table with

10. Create the Automation connection

Note: to create an Azure AD application, AD App Credential and AD Service Principal, it requires Application Developer Role, but works with Application administrator or GLOBAL ADMIN. Non-administrator users in Azure AD tenant can register AD applications if the Azure AD tenant's Users can register applications option on the User settings page is set to Yes.

Below is the script. Start a PowerShell console windows with elevated user rights and navigate to the folder that contains the script.

$ConnectionAssetName = "AzureRunAsConnection"

if (!(Get-AzAutomationConnection -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $ConnectionAssetName -ErrorAction SilentlyContinue)) {
    $CertifcateAssetName = "AzureRunAsCertificate"
    $CertificateName = $AutoAcctName + '-' + $CertifcateAssetName
    if (!( Get-ChildItem Cert:\LocalMachine\My | Where-Object DnsNameList -Contains $certificateName -ErrorAction SilentlyContinue)) {
        $SelfSignedCertPlainPassword = "1@3$5^7*9)"
        $PfxCertPathForRunAsAccount = Join-Path $env:TEMP ($CertificateName + ".pfx")
        $PfxCertPlainPasswordForRunAsAccount = $SelfSignedCertPlainPassword
        $CerCertPathForRunAsAccount = Join-Path $env:TEMP ($CertificateName + ".cer")
        $SelfSignedCertNoOfMonthsUntilExpired = 12
    
        $Cert = New-SelfSignedCertificate -DnsName $certificateName `
            -CertStoreLocation cert:\LocalMachine\My `
            -KeyExportPolicy Exportable `
            -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider" `
            -NotAfter (Get-Date).AddMonths($selfSignedCertNoOfMonthsUntilExpired) `
            -HashAlgorithm SHA256
    
        $CertPassword = ConvertTo-SecureString $selfSignedCertPlainPassword -AsPlainText -Force
    
        Export-PfxCertificate -Cert ("Cert:\localmachine\my\" + $Cert.Thumbprint) -FilePath $PfxCertPathForRunAsAccount -Password $CertPassword -Force | Write-Verbose
    
        Export-Certificate -Cert ("Cert:\localmachine\my\" + $Cert.Thumbprint) -FilePath $CerCertPathForRunAsAccount -Type CERT | Write-Verbose
    }
    $ApplicationDisplayName = "MyTest-CUS-AutoAcct01-AppId-DisplayName"
    if (!(Get-AzADApplication -DisplayName $ApplicationDisplayName)) {
        $PfxCert = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList @($PfxCertPathForRunAsAccount, $PfxCertPlainPasswordForRunAsAccount)
    
        $keyValue = [System.Convert]::ToBase64String($PfxCert.GetRawCertData())
        $keyId = (New-Guid).Guid 
        
        $Application = New-AzADApplication -DisplayName $ApplicationDisplayName -HomePage ("http://" + $ApplicationDisplayName) -IdentifierUris ("http://" + $keyId)
    
        $ApplicationCredential = New-AzADAppCredential -ApplicationId $Application.ApplicationId -CertValue $keyValue -StartDate $PfxCert.NotBefore -EndDate $PfxCert.NotAfter
    
        $ServicePrincipal = New-AzADServicePrincipal -ApplicationId $Application.ApplicationId
 
        $GetServicePrincipal = Get-AzADServicePrincipal -ObjectId $ServicePrincipal.Id
    
        #Sleep here for a few seconds to allow the service principal application to become active (ordinarily takes a few seconds)
        Start-Sleep -s 15
        
        if (!(Get-AzRoleAssignment -ServicePrincipalName $Application.ApplicationId -ErrorAction SilentlyContinue)) {
            $NewRole = New-AzRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $Application.ApplicationId -ErrorAction SilentlyContinue
            $Retries = 0;
            While ($null -eq $NewRole -and $Retries -le 6) {
                Start-Sleep -s 10
                New-AzRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $Application.ApplicationId | Write-Verbose -ErrorAction SilentlyContinue
                $NewRole = Get-AzRoleAssignment -ServicePrincipalName $Application.ApplicationId -ErrorAction SilentlyContinue
                $Retries++;
            }
        }
        $ApplicationId = $Application.ApplicationId.ToString()
    }
    # Create the Automation certificate asset
    $CertPassword = ConvertTo-SecureString $PfxCertPlainPasswordForRunAsAccount -AsPlainText -Force
    if (!(Get-AzAutomationCertificate -ResourceGroupName $RGName `
                -AutomationAccountName $AutoAcctName `
                -Name $CertifcateAssetName `
                -ErrorAction SilentlyContinue)) {
        New-AzAutomationCertificate -ResourceGroupName $RGName `
            -AutomationAccountName $AutoAcctName `
            -Path $PfxCertPathForRunAsAccount `
            -Name $CertifcateAssetName `
            -Password $CertPassword `
            -Exportable:$true
    }
    # Populate the ConnectionFieldValues
    $SubscriptionInfo = Get-AzSubscription -SubscriptionId $SubscriptionId
    $TenantID = $SubscriptionInfo | Select-Object TenantId -First 1
    $Thumbprint = $PfxCert.Thumbprint
    $ConnectionFieldValues = @{"ApplicationId" = $ApplicationId; "TenantId" = $TenantID.TenantId; "CertificateThumbprint" = $Thumbprint; "SubscriptionId" = $SubscriptionId }
    $ConnectionTypeName = "AzureServicePrincipal"
    
    New-AzAutomationConnection -ResourceGroupName $RGName `
        -AutomationAccountName $AutoAcctName `
        -Name $ConnectionAssetName `
        -ConnectionTypeName $ConnectionTypeName `
        -ConnectionFieldValues $ConnectionFieldValues
}

Step 5: Test creating a Automation variable

$MyTestVariableName = "AutoAcctName"
$MyTestVariableValue = $AutoAcctName
if (!(Get-AzAutomationVariable -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $MyTestVariableName -ErrorAction SilentlyContinue)) {
    New-AzAutomationVariable -ResourceGroupName $RGName `
        -AutomationAccountName $AutoAcctName `
        -Name $MyTestVariableName `
        -Value $MyTestVariableValue `
        -Encrypted $false
}

Step 6: Test creating an encrypted Automation variable

$MyTestVariableName = "MyTestPassWord"

$MyTestVariableValue = "The is an encrypted string"

if (!(Get-AzAutomationVariable -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $MyTestVariableName -ErrorAction SilentlyContinue)) {
    New-AzAutomationVariable -ResourceGroupName $RGName `
        -AutomationAccountName $AutoAcctName `
        -Name $MyTestVariableName `
        -Value $MyTestVariableValue `
        -Encrypted $true
}

Step 7: Test creating an Automation runbook

$MyTestRunbookName = "MyTest-Runbook01"

if (!(Get-AzAutomationRunbook -ResourceGroupName $RGName -AutomationAccountName $AutoAcctName -Name $MyTestRunbookName -ErrorAction SilentlyContinue)) {
    New-AzAutomationRunbook -ResourceGroupName $RGName `
        -AutomationAccountName $AutoAcctName `
        -Name $MyTestRunbookName `
        -Type PowerShell
}