Skip to main content

Azure Batch

The azurebatch executor runs pipeline tasks using Azure Batch, a managed compute service that runs containerized workloads in Azure. It submits each task as an Azure Batch task.

To enable this executor, set process.executor = 'azurebatch' in the nextflow.config file. You can launch the pipeline from a local machine or an Azure VM. Every pipeline process must specify a container image using the container directive. The pipeline work directory must be in an Azure Blob Storage container.

Use the following process directives to control resource requests and other task characteristics:

See Azure Blob Storage for how to configure Azure credentials and storage access. See the azure config scope for the full list of configuration options.

tip

For automated Azure infrastructure setup, consider using Seqera Platform, which can create Azure Batch compute environments for you.

Overview​

Nextflow integrates with Azure Batch by mapping its execution model to the Azure Batch structure. Nextflow runs tasks on compute nodes within an Azure Batch pool, which is a collection of virtual machines that can scale up or down based on an autoscale formula.

note

In Azure Batch, a job is not a unit of work like in other batch schedulers. It is a container for tasks that is bound to a single pool, similar to a job queue. The task is the unit of work. Nextflow creates one Azure Batch job for each combination of process and pool, and submits each Nextflow task as an Azure Batch task in that job.

Nextflow manages these pools dynamically. You can assign processes to specific pools using the process queue directive. Nextflow creates the pool if it doesn't exist and azure.batch.allowPoolCreation is set to true. Alternatively, with autoPoolMode, Nextflow automatically creates pools based on the CPU and memory requirements defined in your processes.

Each task first downloads its input files from Azure Blob Storage to its assigned compute node. It then runs the process script. Finally, it uploads any output files back to Azure Blob Storage.

Nextflow marks the corresponding Azure Batch jobs as terminated when the pipeline completes. Compute pools configured for auto-scaling scale down according to their formula to minimize costs. You can also configure Nextflow to delete these node pools after the pipeline completes.

Getting started​

  1. Create an Azure Batch account in the Azure portal.

  2. Increase the quotas in your Azure Batch account to meet the needs of your pipeline. Quotas limit the number of pools, CPUs, and jobs you can create.

  3. Create a storage account and a blob container in the same region as the Batch account.

  4. Make sure that every process in your pipeline specifies a container image with the container directive.

  5. Configure your Azure credentials. See Authentication and Required permissions for details.

  6. Add the following settings to your Nextflow configuration:

    • Set process.executor to azurebatch.
    • Set azure.storage.accountName to your storage account name.
    • Set azure.batch.accountName and azure.batch.location to your Batch account name and location.
    • Set workDir to a Blob Storage path, for example, az://<blob-container>/work.

  7. Launch your pipeline with this configuration:

    nextflow run <pipeline-name>

For example:

process {
    executor = 'azurebatch'
    container = 'quay.io/biocontainers/salmon'
}

workDir = 'az://<blob-container>/work'

azure {
    storage {
        accountName = '<storage-account-name>'
    }
    batch {
        location = 'eastus'
        accountName = '<batch-account-name>'
        autoPoolMode = true
        allowPoolCreation = true
    }
}
warning

The work directory must contain at least one subdirectory. Do not use only the container name, for example, az://<blob-container>.

tip

Nextflow automatically transfers any input data not stored in Azure Blob Storage to the pipeline work directory. Use this feature with caution to avoid unnecessary data transfers.

tip

You can list Azure regions with:

az account list-locations -o table

Quotas​

Azure Batch enforces quotas on resources like pools, jobs, and VMs.

  • Pools: Azure Batch limits the number of pools that can exist at one time. Nextflow throws an error if it attempts to create a pool when this quota is exhausted. To proceed, delete existing pools.

  • Active jobs: Azure Batch limits the number of active jobs. Terminate completed jobs. If you reach this limit, Nextflow throws an error. Terminate or delete jobs before you run additional pipelines.

  • VM cores: Each Azure VM family has a maximum core quota. If you request a VM family size for which you lack sufficient core quota, any jobs and tasks that Nextflow creates remain pending because the quota prevents the pool from scaling. This can cause your pipeline to hang indefinitely.

To increase your quota, open your Batch account in the Azure portal, select Quotas in the left-hand menu, and request an increase.

Pools​

Nextflow supports two approaches for managing Batch pools: auto pools and named pools.

warning

Clean up the Batch pools or use auto-scaling to avoid additional charges in the Batch account.

Auto pools​

When autoPoolMode is enabled, Nextflow automatically creates pools of compute nodes appropriate for your pipeline.

Nextflow uses the cpus and memory directives to find the smallest machine type in the Azure machine family, specified by the machineType directive, that satisfies the requested resources. If you specify no options, Nextflow uses one compute node of type Standard_D4a_v4.

To use autoPoolMode, enable it in the azure.batch config scope:

azure {
    batch {
        autoPoolMode = true
        allowPoolCreation = true
    }
}

process {
    machineType = 'Standard_D4_v3'
    cpus = 4
    memory = 16.GB
}

You can specify multiple machine families per process using glob patterns in the machineType directive:

// D or E v5 machines with data disk
process.machineType = 'Standard_D*d_v5,Standard_E*d_v5'

For example, the following process creates a pool of Standard_E16d_v5 machines when using autoPoolMode (see the following notes on task packing):

process EXAMPLE_PROCESS {
    machineType "Standard_E*d_v5"
    cpus 8
    memory 8.GB

    script:
    """
    echo "cpus: ${task.cpus}"
    """
}

Additional notes:

  • When the machine type is a pattern, Nextflow selects larger machines to enable task packing: 4x the requested resources for tasks using 4 or fewer CPUs, and 2x for tasks using 8 or fewer CPUs.
  • To override this behavior, specify an exact machine type (for example, Standard_E2d_v5).
  • Patterns also support regular expression character classes, which you can use to exclude machine types (for example, Standard_*[^p]_v* excludes ARM-based machines).
  • Nextflow does not remove the pool when the pipeline terminates unless you enable azure.batch.deletePoolsOnCompletion in your Nextflow configuration file.

Named pools​

Use the queue directive to specify the ID of the Azure Batch compute pool for a process.

When azure.batch.allowPoolCreation = true is set in your configuration, Nextflow creates the pools on demand. Otherwise, the pool must be available in the Batch environment when Nextflow submits tasks.

You can configure each pool separately in the Nextflow configuration. For example:

azure {
    batch {
        pools {
            small {
                vmType = 'Standard_D2_v2'
                vmCount = 5
            }
            large {
                vmType = 'Standard_D8_v3'
                vmCount = 2
            }
        }
    }
}

process {
    withLabel: small_jobs {
        queue = 'small'
    }
    withLabel: large_jobs {
        queue = 'large'
    }
}

This example defines the configuration for two node pools, small and large. The small pool provisions 5 compute nodes of type Standard_D2_v2, and the large pool provisions 2 nodes of type Standard_D8_v3. See the azure configuration scope for the list of available configuration options.

warning

Pool names can contain alphanumeric characters, hyphens, and underscores. Hyphenated names must be quoted, for example, 'pool-1'.

Requirements for existing pools​

When you configure Nextflow to use a pool that already exists in the Batch account, the pool must meet the following requirements:

  • The pool must be declared as dockerCompatible (Container Type property).

  • The task slots per node must match the number of cores for the selected VM. Otherwise, Nextflow returns an error of the form "Azure Batch pool 'ID' slots per node does not match the VM num cores (slots: N, cores: Y)".

  • Unless you are using Fusion, all tasks must have the azcopy command-line tool available in the path. If azure.batch.copyToolInstallMode = 'node', every node must have azcopy available at $AZ_BATCH_NODE_SHARED_DIR/bin/.

Auto-scaling​

Azure Batch can automatically scale pools based on the parameters you define. With automatic scaling, Batch dynamically adds nodes to a pool as task demands increase and removes nodes as task demands decrease.

To enable this feature for pools created by Nextflow, add autoScale = true to the corresponding pool configuration scope. For example:

azure {
    batch {
        pools {
            <pool-name> {
                autoScale = true
                vmCount = 5        // Initial count
                maxVmCount = 50    // Maximum count
            }
        }
    }
}

The default auto-scaling formula initializes the pool with the number of VMs specified by the vmCount option. It dynamically scales up the pool based on the number of pending tasks. The maxVmCount option sets the pool limit. When no jobs are submitted for execution, the formula scales the pool down to zero nodes to minimize costs. The default formula is:

// Get pool lifetime since creation.
lifespan = time() - time("{{poolCreationTime}}");
interval = TimeInterval_Minute * {{scaleInterval}};

// Compute the target nodes based on pending tasks.
$samples = $PendingTasks.GetSamplePercent(interval);
$tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max( $PendingTasks.GetSample(1), avg($PendingTasks.GetSample(interval)));
$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes/2);
targetPoolSize = max(0, min($targetVMs, {{maxVmCount}}));

// For first interval deploy 1 node, for other intervals scale up/down as per tasks.
$TargetDedicatedNodes = lifespan < interval ? {{vmCount}} : targetPoolSize;
$NodeDeallocationOption = taskcompletion;

Use the azure.batch.pools.<pool-name>.scaleFormula setting to specify a custom formula.

Task authentication​

By default, Nextflow creates shared access signature (SAS) tokens for specific containers and passes them to tasks for file operations with Azure Storage. SAS tokens expire after 48 hours by default. To change the expiration time, set azure.storage.tokenDuration in your configuration.

Managed identity​

Added in version 25.10
note

Only available when using Fusion file system.

You can also authenticate to Azure Storage using a managed identity when using Fusion.

To do this:

  1. Create a user-assigned managed identity with the Azure Storage Blob Data Contributor role for your storage account.
  2. Attach the user-assigned managed identity to the node pool manually.
  3. Set azure.batch.poolIdentityClientId to this identity in your configuration.
azure {
    batch {
        poolIdentityClientId = '<managed-identity-client-id>'
    }
}

Each task authenticates as the managed identity, and downloads and uploads files to Azure Storage using these credentials. You can attach more than one managed identity to a pool. Fusion uses the identity specified by azure.batch.poolIdentityClientId.

Task packing​

Each Azure Batch node is allocated a specific number of task slots that determine how many tasks can run concurrently on that node. The number of slots is based on the available CPUs of the virtual machine. Nextflow assigns task slots to each process based on the proportion of total node resources requested through the cpus, memory, and disk directives.

For example, consider a Standard_D4d_v5 machine with 4 vCPUs, 16 GB of memory, and 150 GB local disk:

  • If a process requests cpus 2, memory 8.GB, or disk 75.GB, two task slots are allocated (50% of resources). Two tasks can run concurrently on the node.

  • If a process requests cpus 4, memory 16.GB, or disk 150.GB, four task slots are allocated (100% of resources). One task can run on the node.

Resource overprovisioning can occur if tasks consume more than their allocated share of resources. For example, the node described earlier can become overloaded and fail if a task with cpus 2 uses more than 8 GB of memory or 75 GB of disk space. Specify resource requirements accurately to prevent task failures.

warning

Azure virtual machines come with fixed storage disks that are not expandable. Tasks fail if the tasks running concurrently on a node use more storage than the machine has available.

By default, Docker and containerd store image layers on the OS disk, which can fill up quickly when pulling large container images. On VMs with an attached data disk (for example, Standard_E16ds_v5), you can use a pool start task to relocate Docker and containerd storage to $AZ_BATCH_NODE_ROOT_DIR. This directory uses the larger data disk.

Task containers​

Every process in your pipeline must specify a container to run on Azure Batch.

Nextflow supports both public and private container images. You can pull public images from repositories such as https://community.wave.seqera.io. You must authenticate to use private images. When you use a private registry such as Azure Container Registry (ACR), make sure the Batch pool has access. Provide ACR credentials in the azure.registry configuration scope:

azure {
    registry {
        server = '<registry-server>'     // e.g., 'myregistry.azurecr.io'
        userName = '<registry-user>'
        password = '<registry-password>'
    }
}
tip

Nextflow uses the following environment variables if the registry credentials are not provided in the Nextflow configuration file:

  • AZURE_REGISTRY_USER_NAME: the username for Azure Container Registry authentication
  • AZURE_REGISTRY_PASSWORD: the password for Azure Container Registry authentication

You can use container images from public registries such as Docker Hub without additional configuration, even when a private registry is specified in the azure.registry scope. The Docker runtime on Azure Batch VMs automatically uses the appropriate registry based on the container image specified for the task.

VM images​

When Nextflow creates a pool of compute nodes, it selects:

  • The virtual machine image reference (defines the operating system (OS) and software to be installed on the node).
  • The Batch node agent SKU (a program that runs on each node and provides the interface between the node and the Azure Batch service).

These settings determine the operating system, version, and available features for each compute node in your pool.

By default, Nextflow creates pool nodes suitable for most bioinformatics workloads based on Ubuntu 24.04. You can customize the OS in your pool configuration to meet specific requirements.

The following examples show the default image configuration and a custom image configuration:

Ubuntu 24.04 (default)

azure {
    batch {
        pools {
            <pool-name> {
                sku = "batch.node.ubuntu 24.04"
                offer = "ubuntu-hpc"
                publisher = "microsoft-dsvm"
            }
        }
    }
}

Custom images from an Azure Compute Gallery

To provision pool nodes from a custom VM image published in an Azure Compute Gallery, set virtualMachineImageId to the image version resource ID. When virtualMachineImageId is set, Nextflow ignores publisher and offer, and you must set sku to the Batch node agent SKU ID that matches the image operating system.

azure {
    batch {
        pools {
            <pool-name> {
                virtualMachineImageId = '/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Compute/galleries/<gallery>/images/<image-definition>/versions/<version>'
                sku = 'batch.node.ubuntu 24.04'
            }
        }
    }
}
warning

Custom images require Microsoft Entra authentication (service principal or managed identity). The Azure Batch account identity must have read access to the gallery image.

Image verification​

By default, Nextflow selects only images that Azure Batch has verified. Set allowUnverifiedImages to true to also allow images that Azure Batch lists but has not verified. This setting is ignored when virtualMachineImageId is set.

azure {
    batch {
        pools {
            <pool-name> {
                allowUnverifiedImages = true
            }
        }
    }
}

Virtual networks​

You can configure pools to use a virtual network to connect to your existing network infrastructure.

azure.batch.pools.<pool-name>.virtualNetwork = '<subnet-id>'

The subnet ID must be in the following format:

/subscriptions/<subscription>/resourceGroups/<group>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>

Nextflow can submit tasks to existing pools that are already attached to a virtual network without requiring additional configuration.

warning

Virtual networks require Microsoft Entra authentication (service principal or managed identity).

Start tasks​

Start tasks are optional commands that run when a node joins the pool. Use them to set up the node environment or to run other commands when the node is created.

azure {
    batch {
        pools {
            <pool-name> {
                startTask {
                    script = 'echo "Hello, world!"'
                    privileged = true  // optional, defaults to false
                }
            }
        }
    }
}

Azure file shares​

Azure file shares are fully managed file shares that you can mount to compute nodes. Files are immediately available in the file system, and processes can access them as local files.

The Azure file share must exist in the storage account configured for Blob Storage. You must provide the name of the source Azure file share and mount path (the path where the files are mounted). You can also set additional mount options. See the Azure Files documentation for more information.

For example:

azure {
    storage {
        fileShares {
            references {  // any name of your choice
                mountPath = '/mnt/references'
                mountOptions = '-o vers=3.0,dir_mode=0777,file_mode=0777,sec=ntlmssp'  // optional
            }
        }
    }
}
warning

File shares must be authenticated using a storage account key. Managed identity and service principal are not supported.

Spot instances​

Azure Batch pools can use Spot VMs (also known as low-priority VMs) instead of dedicated VMs to reduce costs. To use Spot VMs, set lowPriority = true in the pool configuration:

azure {
    batch {
        pools {
            <pool-name> {
                lowPriority = true
            }
        }
    }
}

When using autoPoolMode, use auto as the pool name.

Azure can preempt Spot VMs before the task completes. When Azure preempts a node, Azure Batch automatically requeues its running tasks so that they run again when a node becomes available.

Fusion file system​

The Azure Batch executor supports Fusion. With Fusion, you can use Azure Blob Storage as a virtual distributed file system. Fusion optimizes data transfer and speeds up most task I/O operations. When using Fusion, tasks can also authenticate to Azure Storage with a managed identity.

See Azure Batch for more information about configuring Fusion for Azure Batch.

On this Page