Daniel Marques

Posted on May 29 • Edited on Jun 1

How to create custom Azure DevOps Pipelines that autoscale with Virtual Machine Scale Sets (VMSS)

#azure #virtualmachine #devops #automation

Microsoft-hosted Azure DevOps pipelines have some limitations, such as not being able to access Azure resources in private networks or having a disk size limit of 10GB. Fortunately, you can work around these by using custom pipelines. One effective approach is to use Virtual Machine Scale Sets (VMSS), which I’ll explain in detail in this post. The source code is available on my GitHub. You can also read the official comparison between VMSS and Microsoft-hosted agents here.

A VMSS is an Azure compute resource that lets you deploy and manage a group of identical virtual machines at scale. In Azure Pipelines, VMSS can host custom agents and automatically scale the number of build and deployment agents based on workload. This provides efficient resource usage, access to private networks, and more control over your build environment compared to Microsoft-hosted agents.

For this post, we’ll consider a scenario where you need to access an Azure Key Vault and it are only visible for specific networks.

The picture below illustrates that.

We’ll achieve this by following these steps:

Install Packer and Azure CLI
Create a VM image that will be used in the VMSS
Create the VMSS
Create a service principal
Configure the VMSS in Azure DevOps
Allow VMSS to access Azure Key Vault

1. Install Packer and Azure CLI

First, install Packer and the Azure CLI using the official guides:

Next, install the Packer Azure plugin:

packer plugins install github.com/hashicorp/azure

If you are running in Windows, you can execute packer in WSL.

2. Create a VM image that will be used in the VMSS

Azure provides several base VM images, but they may not include all the tools you need (for example, Ubuntu images do not come with the Azure CLI pre-installed). To create a custom image, we will use Packer to automate the process.

You’ll need a Packer configuration file and two scripts: one to install your required software (such as the Azure CLI) and another to deprovision the VM, removing machine-specific data. Packer will output a reusable VM image in your chosen region and resource group.

After preparing your files, build the image (replace my-image as needed):

packer build -var 'output_image_name=my-image' modified-ubuntu-image.pkr.hcl

Key points about these files:

The azure-arm / source_image section defines the temporary VM details used by Packer.
The output_image_name variable makes the image name configurable.
The use_azure_cli_auth option uses authentication from the az login command.
Choose the architecture (ARM or Intel) based on your workload; ARM VMs are often cheaper if you don't need x64 compatibility (like running x64 docker images).
The install-requirements.sh script installs required tools, such as the Azure CLI, without using sudo commands.
The deprovision.sh script removes machine-specific data and credentials, ensuring each VM created from the image starts in a clean, secure state.
The execute_command tells Packer to run scripts as root (sudo -E sh), so you don't need to add sudo commands inside your install script.

File: modified-ubuntu-image.pkr.hcl

variable "output_image_name" {
  type    = string
  default = ""
}

source "azure-arm" "source_image" {
  azure_tags = {
    dept = "Engineering"
    task = "Image deployment"
  }
  use_azure_cli_auth                = true
  image_offer                       = "ubuntu-24_04-lts"
  image_publisher                   = "Canonical"
  image_sku                         = "server"
  location                          = "South Central US"
  managed_image_name                = var.output_image_name
  managed_image_resource_group_name = "my-resource-group"
  os_type                           = "Linux"
  vm_size                           = "Standard_D2s_v6"
}

build {
  sources = ["source.azure-arm.source_image"]

  provisioner "shell" {
    execute_command = "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'"
    scripts          = ["./install-requirements.sh", "./deprovision.sh"] 
  }
}

File: deprovision.sh

/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync

You can also automate this process using an Azure DevOps pipeline with the file create-vmss-image.yaml. The picture below shows the pipeline execution result followed by the resource created in Azure Portal.

3. Create the VMSS

Create the VMSS using the Azure CLI command below. Update the variables as needed for your environment:

vmss_name=my-azure-devops-pool
vmss_image_name=my-image
resource_group=my-resource-group
vm_size=Standard_DC1s_v3
vm_storage=Standard_LRS

az vmss create \
    --name ${vmss_name} \
    --resource-group ${resource_group} \
    --image ${vmss_image_name} \
    --vm-sku ${vm_size} \
    --storage-sku ${vm_storage} \
    --authentication-type SSH \
    --generate-ssh-keys \
    --instance-count 0 \
    --disable-overprovision \
    --upgrade-policy-mode manual \
    --single-placement-group false \
    --platform-fault-domain-count 1 \
    --load-balancer "" \
    --orchestration-mode Uniform

Tips:

Set the initial scaling to 0 (--instance-count 0); Azure DevOps will scale the VMSS based on pipeline demand.
Choosing the right VM size is important for cost and performance. You may want to select any available size initially and adjust it later in the Azure Portal. The image below shows VM sizes ordered by cost — make sure that you select a size with the same architecture as your custom image.

You can also automate this process using an Azure DevOps pipeline with the file create-vmss.yaml. The picture below shows the pipeline execution result followed by the VMSS and its private network created in Azure Portal.

4. Create a service principal

To enable Azure DevOps to manage your VMSS, you need a technical user known as a service principal. In many organizations, this is created by the corporate IT administrator team. If that's your case, you can skip to the next section.

If you need to create it yourself, follow the official guide to register an app. For this setup, register with the option Accounts in this organizational directory only.

After registration, create credentials for the service principal by generating a client secret as described here.

5. Configure the VMSS in Azure DevOps

With your service principal ready, assign it the Contributor role on your VMSS. This allows Azure DevOps to scale the VMSS up and down as needed for your pipelines.

Next, in Azure DevOps, create a service connection using Azure Resource Manager and enter the service principal details. Follow the steps in the official documentation.

Once the service connection is set up, create an agent pool in Azure DevOps. This pool name will be referenced in your pipeline YAML files.

To verify your setup, create a simple pipeline that runs a command on the VMSS agent. For example, use az --version to confirm the Azure CLI is installed. Make sure the job uses the VMSS agent pool you created. The screenshot below shows a successful test run.

6. Allow VMSS to access Azure Key Vault

First, configure the VMSS to use a system-assigned identity.

Then assign the Key Vault Administrator role to your user and the system-assigned identity on your Key Vault.

Next, make sure to disable public access and attach the Key Vault to the same network as the VMSS, as illustrated below.

Finally, create a pipeline using the VMSS agent pool and paste the following command line code:

az login --identity

az keyvault key create \
  --vault-name my-private-key-vault \
  --name my-pipeline-key \
  --protection software \
  --kty RSA \
  --size 2048