Autoscale worker pools based on Prometheus metrics

This tutorial describes how to autoscale your Cloud Run worker pools based on Prometheus metrics using Cloud Run External Metrics Autoscaling (CREMA).

The CREMA autoscaler service performs a ratio-based calculation using data from Prometheus. The autoscaler service dynamically adjusts the instance count to ensure your worker pool has the right amount of resources for the current workload. CREMA calculates the CPU utilization of your worker pool over a specific time period and compares it against your configured threshold to adjust instances.

Objectives

In this tutorial, you will:

Deploy a Cloud Run worker pool to run a background workload that reports utilization metrics to Google Cloud managed service for Prometheus.
Deploy the autoscaler CREMA service to dynamically scale the worker pool based on the Prometheus metrics.
Test your CREMA service by observing service logs and verifying instance count changes in the Google Cloud console.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Cloud Run, Parameter Manager, Artifact Registry, Cloud Build, and Cloud Monitoring APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.
Enable the APIs

Install and initialize the gcloud CLI.
Update components:
```
gcloud components update
```

Set the following configuration variables for CREMA used in this tutorial:

export PROJECT_ID=PROJECT_ID
export REGION=us-central1
export CREMA_SA_NAME=crema-service-account
export CONSUMER_SA_NAME=consumer-service-account
export CONSUMER_WORKER_POOL_NAME=worker-pool-consumer
export CREMA_SERVICE_NAME=my-crema-service

Replace PROJECT_ID with the ID of your Google Cloud project.

Set your project ID by running the following command:
```
gcloud config set project $PROJECT_ID
```
You incur charges for your Cloud Run scaling service based on how often you trigger scaling. For more information, estimate costs with the pricing calculator.

Required roles

To get the permissions that you need to complete the tutorial, ask your administrator to grant you the following IAM roles on your project:

Artifact Registry Repository Administrator (roles/artifactregistry.repoAdmin)
Cloud Build Editor (roles/cloudbuild.builds.editor)
Cloud Run Admin (roles/run.admin)
Create Service Accounts (roles/iam.serviceAccountCreator)
Service Account User (roles/iam.serviceAccountUser)
Service Usage Consumer (roles/serviceusage.serviceUsageConsumer)
Parameter Manager Admin (roles/parametermanager.admin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create custom service accounts

This tutorial requires the following two service accounts with minimum permissions required to use the provisioned resources:

Consumer service account: identity for the worker pool that runs a background workload. Run the following command to create the consumer service account:
```
gcloud iam service-accounts create $CONSUMER_SA_NAME \
  --display-name="Consumer service account"
```
CREMA service account: identity for the autoscaler. Run the following command to create the CREMA service account:
```
gcloud iam service-accounts create $CREMA_SA_NAME \
  --display-name="CREMA service account"
```

Grant additional permissions to your custom service accounts

To scale the worker pool, grant the following permissions on the custom service accounts:

Grant your CREMA service account permission to read from the Parameter Manager:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/parametermanager.parameterViewer"

Grant your CREMA service account the permission to scale the worker pool:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/run.developer"

Grant your CREMA service account the service account user role:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

Grant your CREMA service account permission to view metrics:

 gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
   --role="roles/monitoring.viewer"

Grant your CREMA service account permission to write metrics:

 gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
   --role="roles/monitoring.metricWriter"

Deploy a Cloud Run worker pool

Deploy a worker pool with 0 instances for CREMA to scale up:

gcloud beta run worker-pools deploy $CONSUMER_WORKER_POOL_NAME \
  --image us-docker.pkg.dev/cloudrun/container/worker-pool:latest \
  --instances 0 \
  --region $REGION \
  --memory 4G \
  --cpu 4 \
  --service-account="$CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com"

Deploy the autoscaler CREMA service

Deploy the CREMA service to autoscale your worker pool based on Prometheus metrics.

Configure the autoscaler

This tutorial uses the Parameter Manager to store the YAML configuration file for CREMA.

Create a parameter in the Parameter Manager to store parameter versions for CREMA:

PARAMETER_ID=crema-config
PARAMETER_REGION=global
gcloud parametermanager parameters create $PARAMETER_ID --location=$PARAMETER_REGION --parameter-format=YAML

In your root directory, create a YAML file, my-crema-config.yaml to define the autoscaler configuration. Set the autoscaling threshold to 50% CPU utilization:

apiVersion: crema/v1
kind: CremaConfig
spec:
  pollingInterval: 30
  triggerAuthentications:
    - metadata:
        name: google-crema-auth
      spec:
        podIdentity:
          provider: gcp
  scaledObjects:
    - spec:
        scaleTargetRef:
          name: projects/PROJECT_ID/locations/us-central1/workerPools/worker-pool-consumer
        minReplicaCount: 1
        maxReplicaCount: 20
        triggers:
          - type: prometheus
            metadata:
              serverAddress: https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus
              threshold: "0.5"
              query: |
                histogram_quantile(
                  0.50,
                  sum by (le) (
                    increase(
                      run_googleapis_com:container_cpu_utilizations_bucket{
                        monitored_resource="cloud_run_worker_pool",
                        worker_pool_name="worker-pool-consumer",
                        location="us-central1",
                        project_id="PROJECT_ID"
                      }[2m]
                    )
                  )
                )
            authenticationRef:
              name: google-crema-auth
        advanced:
          horizontalPodAutoscalerConfig:
            behavior:
              scaleDown:
                stabilizationWindowSeconds: 300

Replace PROJECT_ID with the Google Cloud project ID.

Upload your local YAML file as a new parameter version:

LOCAL_YAML_CONFIG_FILE=my-crema-config.yaml
PARAMETER_VERSION=1

gcloud parametermanager parameters versions create $PARAMETER_VERSION \
  --location=$PARAMETER_REGION \
  --parameter=$PARAMETER_ID \
  --payload-data-from-file=$LOCAL_YAML_CONFIG_FILE

Run the following command to verify your parameter addition is successful:
```
gcloud parametermanager parameters versions list \
--parameter=$PARAMETER_ID \
--location=$PARAMETER_REGION
```
You should see the parameter path, such as projects/PROJECT_ID/locations/global/parameters/crema-config/versions/1.

Deploy the service to scale your workloads

To deploy the service to scale your worker pool, run the following command with a prebuilt container image:

CREMA_CONFIG_PARAM_VERSION=projects/$PROJECT_ID/locations/$PARAMETER_REGION/parameters/$PARAMETER_ID/versions/$PARAMETER_VERSION
IMAGE=us-central1-docker.pkg.dev/cloud-run-oss-images/crema-v1/autoscaler:1.0

gcloud beta run deploy $CREMA_SERVICE_NAME \
  --image=${IMAGE} \
  --region=${REGION} \
  --service-account="${CREMA_SA_NAME}" \
  --no-allow-unauthenticated \
  --no-cpu-throttling \
  --base-image=us-central1-docker.pkg.dev/serverless-runtimes/google-24/runtimes/java25 \
  --labels=created-by=crema \
  --set-env-vars="CREMA_CONFIG=${CREMA_CONFIG_PARAM_VERSION},OUTPUT_SCALER_METRICS=True"

Test your autoscaling service

To verify your autoscaling service is working correctly, check the Logs tab of the Cloud Run service. The CREMA autoscaler service scales up the consumer worker instances from 0.

You should see the following logs in your service's logs each time metrics are refreshed:

[INFO] [METRIC-PROVIDER] Starting metric collection cycle
[INFO] [METRIC-PROVIDER] Successfully fetched scaled object metrics ...
[INFO] [METRIC-PROVIDER] Sending scale request ...
[INFO] [SCALER] Received ScaleRequest ...
[INFO] [SCALER] Current instances ...
[INFO] [SCALER] Recommended instances ...

Cloud Run labels each log message with the component that emitted it.

Clean up

To avoid additional charges to your Google Cloud account, delete all the resources you deployed with this tutorial.

Delete the project

If you created a new project for this tutorial, delete the project. If you used an existing project and need to keep it without the changes you added in this tutorial, delete resources that you created for the tutorial.

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

In the Google Cloud console, go to the Manage resources page.
Go to Manage resources
In the project list, select the project that you want to delete, and then click Delete.
In the dialog, type the project ID, and then click Shut down to delete the project.

Delete tutorial resources

Delete the Cloud Run service you deployed in this tutorial. Cloud Run services don't incur costs until they receive requests.

To delete your Cloud Run service, run the following command:
```
gcloud run services delete SERVICE-NAME
```
Replace SERVICE-NAME with the name of your service.

You can also delete Cloud Run services from the Google Cloud console.
Remove the gcloud default region configuration you added during tutorial setup:
```
 gcloud config unset run/region
```
Remove the project configuration:
```
 gcloud config unset project
```
Delete other Google Cloud resources created in this tutorial:
- Delete the Cloud Run worker pool
- Delete the worker pool container image from Artifact Registry
- Delete the CREMA service
- Delete the service accounts
- Delete the parameters in Parameter Manager

What's next

Learn more about Cloud Run worker pools.
Explore other Cloud Run demos, tutorials, and samples.
Configure other KEDA scalers with CREMA.