Create AAA Hybrid Worker Group On VMSS With PowerShell - Part 3

Part 1, Part 2, Part4

Step 15: Enable Termination Notifications on VMSS

This is a key feature which actually makes VMSS suitable for workloads like Automation Hybrid Runbook Workers or Azure Data Factory self-hosed runtime nodes. With this feature enable, a shutdown script on VMSS instances can be configure to run automatically upon receiving a termination notification to unregister the VMSS instance from Hybrid Worker Group before the instance being turned off.

# Enable termination notifications on an existing scale set with a delay 
# timeout of 5 minutes

Update-AzVmss -ResourceGroupName $RGName `
    -VMScaleSetName $VMSSName `
    -Overprovision $false
    -TerminateScheduledEvents $true `
    -TerminateScheduledEventNotBeforeTimeoutInMinutes 5
    
$MyTestVMSSInstances = Get-AzVmssVM -ResourceGroupName $RGName -VMScaleSetName $VMSSName

Get-AzVmssVM -ResourceGroupName $RGName -VMScaleSetName $VMSSName -InstanceId 0

Step 16: Add NSG to an existing VMSS and apply rules to existing instances (optional)

The NSG can also be added to the VMSS if it's not implemented during VMSS creation

# $VMSSNICs = $MyTestVMSS.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations

$VMSSNICs[0].NetworkSecurityGroup = $MyTestNICNSG.Id
Update-AzVmss -ResourceGroupName $RGName -Name $VMSSName -VirtualMachineScaleSet $MyTestVMSS

# bring the existing VMs up-to-date with the latest changes on the 
# scale set 

$MyTestVMSSInstances | Update-AzVmssInstance -VMScaleSetName $VMSSName

Step 17: Connect to VMSS instances through load balancer

# Get the load balancer object

$LB = Get-AzLoaDPAlancer -ResourceGroupName $RGName -Name $LBName

# View the list of inbound NAT rules

Get-AzLoaDPAlancerInboundNatRuleConfig -LoaDPAlancer $LB | Select-Object Name, Protocol, FrontendPort, BackendPort

# Create a remote connection to the first VM instance. Specify the load
# balancer's public IP address and port number mapping to the 
# VM instance, as shown from the preceding commands.

$LBPubIPAddress = Get-AzPublicIpAddress -Name $PubIPAddressName | Select-Object IpAddress

mstsc /v $LBPubIPAddress:50000

Step 18: Create an Azure Key Vault

Azure automation runbooks need the AzureRunAsCertificate available on hybrid runbook workers (VMSS instances in our case here). We can use The Add-AzVmssSecret cmdlet adds a secret to the Virtual Machine Scale Set (VMSS), but the secret must be stored in an Azure Key Vault. So first,

$KVName = "MyTest-CUS-KV01"

if (!(Get-AzKeyVault -ResourceGroupName $RGName -Name $KVName -ErrorAction SilentlyContinue)) {
    $MyTestKV = New-AzKeyVault -ResourceGroupName $RGName -Name $KVName -Location $Location
}

# Allow VMs (VMSS instances) to retrieve certificates stored as 
# secrets from the Key Vault

Set-AzKeyVaultAccessPolicy -VaultName $KVName -ResourceGroupName $RGName -EnabledForDeployment

Step 19: Import certificate into key vault as a secret

The Automaton PFX certificate file created in the previous step contain a private key password which has to be removed so that it can imported into the key vault. The script below shows how OpenSSL can extract the private key without a password into a separate file then create a new PFX file without a password.

Run commands blow in a Git Bash window or anywhere the command openssl is available.

cd $env:TEMP

# Extract the certificate:

openssl pkcs12 -clcerts -nokeys -in "MyTest-CUS-AutoAcct01-AzureRunAsCertificate.pfx" -out MyTest-CUS-AutoAcct01-AzureRunAsCertificate.crt -password pass:"*" -passin pass:"*"

# Extract the certificate authority key:

openssl pkcs12 -cacerts -nokeys -in "MyTest-CUS-AutoAcct01-AzureRunAsCertificate.pfx" -out MyTest-CUS-AutoAcct01-AzureRunAsCertificate.ca -password pass:"*" -passin pass:"*"

# Extract the private key:

openssl pkcs12 -nocerts -in "MyTest-CUS-AutoAcct01-AzureRunAsCertificate.pfx" -out MyTest-CUS-AutoAcct01-AzureRunAsCertificate-private.key -password pass:"*" -passin pass:"*" -passout pass:"*"

# Remove the passphrase from the private key file:

openssl rsa -in MyTest-CUS-AutoAcct01-AzureRunAsCertificate-private.key -out "TargetFile.Key" -passin pass:"*"

# Create a new input file to generate a PfX file:

cat MyTest-CUS-AutoAcct01-AzureRunAsCertificate-private.key MyTest-CUS-AutoAcct01-AzureRunAsCertificate.crt MyTest-CUS-AutoAcct01-AzureRunAsCertificate.ca > MyTest-CUS-AutoAcct01-AzureRunAsCertificate.pem

# Generate a new PFX file without a password:

openssl pkcs12 -export -nodes -CAfile MyTest-CUS-AutoAcct01-AzureRunAsCertificate.ca -in MyTest-CUS-AutoAcct01-AzureRunAsCertificate.pem -passin pass:"*" -passout pass:"" -out "MyTest-CUS-AutoAcct01-AzureRunAsCertificate-NoPWD.pfx"

Now go back to the PowerShell window and run below script to finally upload the secret to the key vault

$PfxCertPathForRunAsAccountNoPWD = Join-Path $env:TEMP ($CertificateName + "-NoPWD.pfx")

Get-ChildItem $PfxCertPathForRunAsAccountNoPWD

# the following two PS commands need to be run in PowerShell core 6.0+

$PFXcontentBytes = Get-Content $PfxCertPathForRunAsAccountNoPWD -AsByteStream

$CertificateSecretValue = [system.convert]::ToBase64String($PFXcontentBytes)

$SecretValue = ConvertTo-SecureString -String $CertificateSecretValue -AsPlainText -Force

$Expires = (Get-Date).AddYears(2).ToUniversalTime()

$SecretContentType = "application/x-pkcs12"


# Upload the certificate into the key vault as a secret:

$CertInKVSecrets = Set-AzKeyVaultSecret -VaultName $KVName -ContentType $SecretContentType -SecretValue $SecretValue -Name $certificateName -Expires $Expires

# $CertInKVSecrets = Get-AzKeyVaultSecret -VaultName $KVName -Name $CertificateName

Step 19: Add the secret (KV certificate) to VMSS

$CertificateUrl = $CertInKVSecrets.Id
$CertificateStore = "My"

$VMSSCertConfig = New-AzVmssVaultCertificateConfig -CertificateUrl $CertificateUrl -CertificateStore $CertificateStore

$TempVMSSConfig = New-AzVmssConfig -Location $Location

#$MyTestKV = Get-AzKeyVault -ResourceGroupName $RGName -Name $KVName

Add-AzVmssSecret -VirtualMachineScaleSet $TempVMSSConfig -SourceVaultId $MyTestKV.ResourceId -VaultCertificate $VMSSCertConfig
Update-AzVmss -ResourceGroupName $RGName -VMScaleSetName $VMSSName -VirtualMachineScaleSet $TempVMSSConfig

Step 20: Verify the certificate is available on the VMSS instances

$MyTestVMSS = Get-AzVmss -VMScaleSetName $VMSSName -ResourceGroupName $RGName

$MyTestVMSS.VirtualMachineProfile.OsProfile.Secrets | Select-Object -ExpandProperty VaultCertificates

Logon to the instances and browse the certificate stores looking for the certificate

$LB = Get-AzLoaDPAlancer -ResourceGroupName $RGName -Name $LBName

$LB.InboundNatRules | Select-Object FrontendPort, BackendPort

$LBPubIPAddress = (Get-AzPublicIpAddress -Name $PubIPAddressName).IpAddress

# Connect to the VMs:
mstsc /v $LBPubIPAddress:50000

Step 21: Create and configure an Azure Log Analytics Workspace

The Hybrid Runbook Worker role depends on an Azure Monitor Log Analytics workspace to install and configure the role. Log Analytics was previously called Operational Insights, which is why it is the name used in the cmdlets.

$LAWorkspaceName = "MyTest-CUs-LAWorkspace01"

if (!(Get-AzOperationalInsightsWorkspace -ResourceGroupName $RGName -Name $LAWorkspaceName -ErrorAction SilentlyContinue)) {
    $MyLAWorkspace = New-AzOperationalInsightsWorkspace -Location $Location -Name $LAWorkspaceName -Sku standard -ResourceGroupName $RGName
}

Enable the Azure Automation solution in the Log Analytics workspace by running the following command in an elevated PowerShell command prompt

Set-AzOperationalInsightsIntelligencePack -ResourceGroupName $RGName -WorkspaceName $LAWorkspaceName -IntelligencePackName "AzureAutomation" -Enabled $true

Get-AzOperationalInsightsIntelligencePack -ResourceGroupName $RGName -WorkspaceName $LAWorkspaceName | Where-Object Name -eq "AzureAutomation"

Step 22: Retrieve workspace ID and workspace key

The Hybrid Runbook Worker role requires the Log Analytics agent for the supported Windows operating system. The extension requires the workspace ID and workspace key from the target Log Analytics workspace. These information can also be found in the settings for the workspace in the Azure portal. Because the workspace key should be treated as sensitive data, it should be stored in a protected setting configuration. Azure VM extension protected setting data is encrypted and only decrypted on the target virtual machine. Note that workspaceId and workspaceKey are case-sensitive.

$LAWorkspaceId = ($MyLAWorkspace).CustomerId

$LAWorkspacePrimaryKey = ($MyLAWorkspace | Get-AzOperationalInsightsWorkspaceSharedKey).PrimarySharedKey

Tips: How to remove all the instances from VMSS?

$MyTestVMSS.Sku.Capacity=0

Update-AzVmss -ResourceGroupName $RGName -VMScaleSetName $VMSSName -VirtualMachineScaleSet $MyTestVMSS

Step 23: Add the Log Analytics Agent extension to the scale set model

$PublicSettings = @{"workspaceId" = $LAWorkspaceId }
$ProtectedSettings = @{"workspaceKey" = $LAWorkspacePrimaryKey }
$ExtensionName = "MicrosoftMonitoringAgent"
$ExtensionType = "MicrosoftMonitoringAgent"
$Publisher = "Microsoft.EnterpriseCloud.Monitoring"

# Add the Log Analytics Agent extension to the scale set model

if ($null -ne ($MyTestVMSS.VirtualMachineProfile.ExtensionProfile.Extensions | Where-Object Name -eq $ExtensionName -ErrorAction SilentlyContinue)){
    $MyTestVMSS | Remove-AzVmssExtension -Name $ExtensionName
    Update-AzVmss -ResourceGroupName $RGName -VMScaleSetName $VMSSName -VirtualMachineScaleSet $MyTestVMSS
}

Add-AzVMSSExtension -VirtualMachineScaleSet $MyTestVMSS `
    -Name $ExtensionName `
    -Publisher $Publisher `
    -Type $ExtensionType `
    -TypeHandlerVersion 1.0 `
    -Setting $PublicSettings `
    -ProtectedSetting $ProtectedSettings `
    -Verbose
    
# Update the scale set

Update-AzVmss -ResourceGroupName $RGName -VMScaleSetName $VMSSName -VirtualMachineScaleSet $MyTestVMSS