Azure Blobs Lifecycle Management Part 1: Solution Overview

Part2, Part3, Part4, Part5

Real-world Case:

A blob container contains more than 20 millions database backup files that occupy more than 1PB storage space. A new batch of database backup files is being added to the container every 15 minutes of every hour of every day. The current Long Term Retention only requires to keep:

• Each Weekly Backup for 5 Weeks.

• First Full Backup of the Month for 13 Months.

• First Full Backup of the Year for 7 Years.

This means most of the backup files are not needed for LTR and should be removed for saving storage cost and better blob searching performance.

Task:

Find a solution to automatically remove the old blob files in the container that are retired from Long Term Retention

The solution:

1. Create a blob triggered Azure function(C# on .NET) to automatically assign index tags to the newly created/updated/modified blobs.

2. Create and run some manually triggered Azure functions (C# on .NET) to assign and verify index tags on all the existing backup files in the target container.

3. Create lifecycle management policies on the storage accounts to automatically delete the old blobs basing on the last modified date and the index tags.

Azure storage account lifecycle management policy rules:

• Rule 1: For SQL Server database transaction log or differential backup files, delete the files if the last modified date (created on) is older than 15 days.

• Rule 2: For SQL Server database full backup files, delete the files if the last modified date is older than 35 days and the blob index tag “KeepForLTRMonthly” = “NO” and “KeepForLTRYearly” = “NO”

• Rule 3: For SQL Server database full backup files, delete the files if the last modified date is older than 13 month and the blob index tag “KeepForLTRYearly” = “NO”

• Rule 4: For SQL Server database full backup files, delete the files if the last modified date is older than 7 years

Blob index tagging strategy:

Add Lifecycle Management Rules

Below is the steps of enabling Azure storage account lifecycle management through Azure portal:

Step 1: Locate and click on the target storage account, then click "Lifecycle management" in "Data management" section, then click "Add a rule" at "Lifecycle management" screen as showed blow:

Step2: At the "Add a rule" screen, click on "Details" tab, type in the rule name, for example, "Remove-Retired-TLog". For Rule scope, select "Limit blobs with filters". Blob type is "Block blobs" and Blob subtype is "Base blobs". As showed blow:

Step 3: Click on "Base blobs" tab at the same "Add a rule" screen, then type in 15 for the "Last modified" value as showed below:

Step 4: Click on "Filter set" tab at the same "Add a rule" screen. For "blob prefix", enter the name of the container followed by a forward slash, for example, "pccstguspassvdatabase/". For "Blob index match" add these three key/value pairs:

• Key = "BackupType", Value = "LOG";

• Key = "KeepForLTRMonthly", Value = "NO";

• Key = "KeepForLTRYearly", Value = "NO"

as showed below:

Step 5: Review and double check all the setting on each tabs, make sure no error or typos, then click "Add" on the "Filter set" screen as showed above.