# List cluster activity events Launch stage: GA `POST /api/2.1/clusters/events` Retrieves a list of events about the activity of a cluster. This API is paginated. If there are more events to read, the response includes all the parameters necessary to request the next page of events. API scopes: clusters ## Request body - `cluster_id` (string, required) The ID of the cluster to retrieve events about. Example: `1234-567890-reef123` - `start_time` (int64, optional) The start time in epoch milliseconds. If empty, returns events starting from the beginning of time. Example: `1617238800000` - `end_time` (int64, optional) The end time in epoch milliseconds. If empty, returns events up to the current time. Example: `1619485200000` - `order` (string, optional) The order to list events in; either "ASC" or "DESC". Defaults to "DESC". Possible values: - `DESC` - `ASC` Default: `DESC` - `event_types` (array of string, optional) An optional set of event types to filter on. If empty, all event types are returned. Possible values: - `CREATING` - `STARTING` - `RESTARTING` - `TERMINATING` - `EDITED` - `RUNNING` - `RESIZING` - `NODES_LOST` - `UPSIZE_COMPLETED` - `INIT_SCRIPTS_STARTED` - `INIT_SCRIPTS_FINISHED` - `DID_NOT_EXPAND_DISK` - `EXPANDED_DISK` - `FAILED_TO_EXPAND_DISK` - `DRIVER_HEALTHY` - `DRIVER_NOT_RESPONDING` - `DRIVER_UNAVAILABLE` - `SPARK_EXCEPTION` - `METASTORE_DOWN` - `DBFS_DOWN` - `AUTOSCALING_STATS_REPORT` - `NODE_BLACKLISTED` - `PINNED` - `UNPINNED` - `NODE_EXCLUDED_DECOMMISSIONED` - `ADD_NODES_FAILED` - `AUTOSCALING_BACKOFF` - `AUTOMATIC_CLUSTER_UPDATE` - `AUTOSCALING_FAILED` - `CLUSTER_MIGRATED` - `DECOMMISSION_STARTED` - `DECOMMISSION_ENDED` - `DEFERRED_POLICY_ENFORCEMENT_SCHEDULED` - `DEFERRED_POLICY_ENFORCEMENT_FAILED` - `UC_VOLUME_MISCONFIGURED` - `offset` (int64, optional, Deprecated) Deprecated: use page_token in combination with page_size instead. The offset in the result set. Defaults to 0 (no offset). When an offset is specified and the results are requested in descending order, the end_time field is required. Example: `5` - `limit` (int64, optional, Deprecated) Deprecated: use page_token in combination with page_size instead. The maximum number of events to include in a page of events. Defaults to 50, and maximum allowed value is 500. Default: `50` Constraints: `<= 500` - `page_token` (string, optional) Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of events respectively. If page_token is empty, the first page is returned. - `page_size` (int32, optional) The maximum number of events to include in a page of events. The server may further constrain the maximum number of results returned in a single page. If the page_size is empty or 0, the server will decide the number of results to be returned. The field has to be in the range [0,500]. If the value is outside the range, the server enforces 0 or 500. Default: `50` ## Returns - `events` (array of object, optional) - `cluster_id` (string, required) - `timestamp` (int64, optional) The timestamp when the event occurred, stored as the number of milliseconds since the Unix epoch. If not provided, this will be assigned by the Timeline service. - `type` (string, optional) Possible values: - `CREATING` - `STARTING` - `RESTARTING` - `TERMINATING` - `EDITED` - `RUNNING` - `RESIZING` - `NODES_LOST` - `UPSIZE_COMPLETED` - `INIT_SCRIPTS_STARTED` - `INIT_SCRIPTS_FINISHED` - `DID_NOT_EXPAND_DISK` - `EXPANDED_DISK` - `FAILED_TO_EXPAND_DISK` - `DRIVER_HEALTHY` - `DRIVER_NOT_RESPONDING` - `DRIVER_UNAVAILABLE` - `SPARK_EXCEPTION` - `METASTORE_DOWN` - `DBFS_DOWN` - `AUTOSCALING_STATS_REPORT` - `NODE_BLACKLISTED` - `PINNED` - `UNPINNED` - `NODE_EXCLUDED_DECOMMISSIONED` - `ADD_NODES_FAILED` - `AUTOSCALING_BACKOFF` - `AUTOMATIC_CLUSTER_UPDATE` - `AUTOSCALING_FAILED` - `CLUSTER_MIGRATED` - `DECOMMISSION_STARTED` - `DECOMMISSION_ENDED` - `DEFERRED_POLICY_ENFORCEMENT_SCHEDULED` - `DEFERRED_POLICY_ENFORCEMENT_FAILED` - `UC_VOLUME_MISCONFIGURED` - `details` (object, optional) - `current_num_workers` (int32, optional) The current number of nodes in the cluster. - `target_num_workers` (int32, optional) The targeted number of nodes in the cluster. - `previous_attributes` (object, optional) The cluster attributes before a cluster was edited. - `cluster_name` (string, optional) Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs. - `spark_version` (string, optional) The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of available Spark versions can be retrieved by using the [clusters/sparkVersions](https://docs.databricks.com/api/workspace/clusters/sparkversions) API call. - `spark_conf` (object, optional) An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` respectively. - `aws_attributes` (object, optional) Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used. - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `0` - `availability` (string, optional) Possible values: - `SPOT` - `ON_DEMAND` - `SPOT_WITH_FALLBACK` Default: `SPOT_WITH_FALLBACK` - `zone_id` (string, optional) Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the Databricks deployment. For example, "us-west-2a" is not a valid zone id if the Databricks deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value can be found by using the `List Zones` method. - `instance_profile_arn` (string, optional) Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the Databricks environment by an account administrator. This feature may only be available to certain customer plans. - `spot_bid_price_percent` (int32, optional) The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new `r3.xlarge` spot instance, then the bid price is half of the price of on-demand `r3.xlarge` instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand `r3.xlarge` instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000. Default: `100` - `ebs_volume_type` (string, optional) The type of EBS volumes that will be launched with this cluster. Possible values: - `GENERAL_PURPOSE_SSD` - `THROUGHPUT_OPTIMIZED_HDD` - `ebs_volume_count` (int32, optional) The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These EBS volumes will be mounted at `/ebs0`, `/ebs1`, and etc. Instance store volumes will be mounted at `/local_disk0`, `/local_disk1`, and etc. If EBS volumes are attached, Databricks will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are specified, then the Spark configuration `spark.local.dir` will be overridden. Default: `0` - `ebs_volume_size` (int32, optional) The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096. - `ebs_volume_iops` (int32, optional) If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used. - `ebs_volume_throughput` (int32, optional) If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used. - `azure_attributes` (object, optional) Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used. - `log_analytics_info` (object, optional) Defines values necessary to configure and run Azure Log Analytics agent - `log_analytics_workspace_id` (string, optional) - `log_analytics_primary_key` (string, optional) - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `1` - `availability` (string, optional) Availability type used for all subsequent nodes past the `first_on_demand` ones. Note: If `first_on_demand` is zero, this availability type will be used for the entire cluster. Possible values: - `SPOT_AZURE` - `ON_DEMAND_AZURE` - `SPOT_WITH_FALLBACK_AZURE` Default: `ON_DEMAND_AZURE` - `spot_bid_max_price` (double, optional) The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1. Default: `-1` - `capacity_reservation_group` (string, optional) The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation. Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions: 1. Microsoft.Compute/capacityReservationGroups/read 2. Microsoft.Compute/capacityReservationGroups/deploy/action 3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read 4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action Format: `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}` - `gcp_attributes` (object, optional) Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used. - `use_preemptible_executors` (boolean, optional, Deprecated) This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead. Default: `false` - `google_service_account` (string, optional) If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the Databricks environment by an account administrator. - `boot_disk_size` (int32, optional) Boot disk size in GB - `availability` (string, optional) This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable. Possible values: - `PREEMPTIBLE_GCP` - `ON_DEMAND_GCP` - `PREEMPTIBLE_WITH_FALLBACK_GCP` Default: `ON_DEMAND_GCP` - `zone_id` (string, optional) Identifier for the availability zone in which the cluster resides. This can be one of the following: - "HA" => High availability, spread nodes across availability zones for a Databricks deployment region [default]. - "AUTO" => Databricks picks an availability zone to schedule the cluster on. - A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones. Default: `HA` - `local_ssd_count` (int32, optional) If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to [GCP documentation](https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds) for the supported number of local SSDs for each instance type. - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `1` - `node_type_id` (string, optional) This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the [clusters/listNodeTypes](https://docs.databricks.com/api/workspace/clusters/listnodetypes) API call. - `driver_node_type_id` (string, optional) The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as `node_type_id` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. - `worker_node_type_flexibility` (object, optional) Flexible node type configuration for worker nodes. - `alternate_node_type_ids` (array of string, optional) A list of node type IDs to use as fallbacks when the primary node type is unavailable. - `aws_context_id` (string, optional) The AWS Context ID for EC2 Fleet. When set (non-empty), the value is passed to AWS CreateFleet API to create the EC2 Fleet. - `driver_node_type_flexibility` (object, optional) Flexible node type configuration for the driver node. - `alternate_node_type_ids` (array of string, optional) A list of node type IDs to use as fallbacks when the primary node type is unavailable. - `aws_context_id` (string, optional) The AWS Context ID for EC2 Fleet. When set (non-empty), the value is passed to AWS CreateFleet API to create the EC2 Fleet. - `ssh_public_keys` (array of string, optional) SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name `ubuntu` on port `2200`. Up to 10 keys can be specified. - `custom_tags` (object, optional) Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to `default_tags`. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags - `cluster_log_conf` (object, optional) The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every `5 mins`. The destination of driver logs is `$destination/$clusterId/driver`, while the destination of executor logs is `$destination/$clusterId/executor`. - `dbfs` (object, optional) destination needs to be provided. e.g. `{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `volumes` (object, optional) destination needs to be provided, e.g. `{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `spark_env_vars` (object, optional) An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., `export X='Y'`) while launching the driver and workers. In order to specify an additional set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` - `autotermination_minutes` (int32, optional) Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination. - `enable_elastic_disk` (boolean, optional) Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. **AWS:** Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details. - `init_scripts` (array of object, optional) The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If `cluster_log_conf` is specified, init script logs are sent to `//init_scripts`. - `dbfs` (object, optional, Deprecated) destination needs to be provided. e.g. `{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `file` (object, optional) destination needs to be provided, e.g. `{ "file": { "destination": "file:/my/local/file.sh" } }` - `destination` (string, required) local file destination, e.g. `file:/my/local/file.sh` - `gcs` (object, optional) destination needs to be provided, e.g. `{ "gcs": { "destination": "gs://my-bucket/file.sh" } }` - `destination` (string, required) GCS destination/URI, e.g. `gs://my-bucket/some-prefix` - `abfss` (object, optional) destination needs to be provided, e.g. `abfss://@.dfs.core.windows.net/` - `destination` (string, required) abfss destination, e.g. `abfss://@.dfs.core.windows.net/`. - `workspace` (object, optional) destination needs to be provided, e.g. `{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }` - `destination` (string, required) wsfs destination, e.g. `workspace:/cluster-init-scripts/setup-datadog.sh` - `volumes` (object, optional) destination needs to be provided. e.g. `{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `docker_image` (object, optional) Custom docker image BYOC - `url` (string, optional) URL of the docker image. - `basic_auth` (object, optional) Basic auth with username and password - `username` (string, optional) Name of the user - `password` (string, optional) Password of the user - `instance_pool_id` (string, optional) The optional ID of the instance pool to which the cluster belongs. - `single_user_name` (string, optional) Single user name if data_security_mode is `SINGLE_USER` - `policy_id` (string, optional) The ID of the cluster policy used to create the cluster if applicable. - `enable_local_disk_encryption` (boolean, optional) Whether to enable LUKS on cluster VMs' local disks - `driver_instance_pool_id` (string, optional) The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned. - `workload_type` (object, optional) - `clients` (object, required) defined what type of clients can use the cluster. E.g. Notebooks, Jobs - `notebooks` (boolean, optional) With notebooks set, this cluster can be used for notebooks Default: `true` - `jobs` (boolean, optional) With jobs set, the cluster can be used for jobs Default: `true` - `data_security_mode` (string, optional) Possible values: - `NONE` - `SINGLE_USER` - `USER_ISOLATION` - `LEGACY_TABLE_ACL` - `LEGACY_PASSTHROUGH` - `LEGACY_SINGLE_USER` - `LEGACY_SINGLE_USER_STANDARD` - `DATA_SECURITY_MODE_STANDARD` - `DATA_SECURITY_MODE_DEDICATED` - `DATA_SECURITY_MODE_AUTO` - `runtime_engine` (string, optional) Determines the cluster's runtime engine, either standard or Photon. This field is not compatible with legacy `spark_version` values that contain `-photon-`. Remove `-photon-` from the `spark_version` and set `runtime_engine` to `PHOTON`. If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used. Possible values: - `NULL` - `STANDARD` - `PHOTON` - `kind` (string, optional) Possible values: - `COMPUTE_KIND_UNSPECIFIED` - `CLASSIC_PREVIEW` - `use_ml_runtime` (boolean, optional) This field can only be used when `kind = CLASSIC_PREVIEW`. `effective_spark_version` is determined by `spark_version` (DBR release), this field `use_ml_runtime`, and whether `node_type_id` is gpu node or not. - `is_single_node` (boolean, optional) This field can only be used when `kind = CLASSIC_PREVIEW`. When set to true, Databricks will automatically set single node related `custom_tags`, `spark_conf`, and `num_workers` - `remote_disk_throughput` (int32, optional) If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks. - `total_initial_remote_disk_size` (int32, optional) If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP. - `dependency_mode` (string, optional, Beta) Controls dependency configuration for the cluster. Possible values: - `DEPENDENCY_MODE_UNSPECIFIED` - `DEPENDENCY_MODE_ENVIRONMENTS` - `DEPENDENCY_MODE_CLUSTER_LIBRARIES` - `DEPENDENCY_MODE_AUTO` - `attributes` (object, optional) * For created clusters, the attributes of the cluster. * For edited clusters, the new attributes of the cluster. - `cluster_name` (string, optional) Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs. - `spark_version` (string, optional) The Spark version of the cluster, e.g. `3.3.x-scala2.11`. A list of available Spark versions can be retrieved by using the [clusters/sparkVersions](https://docs.databricks.com/api/workspace/clusters/sparkversions) API call. - `spark_conf` (object, optional) An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` respectively. - `aws_attributes` (object, optional) Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used. - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `0` - `availability` (string, optional) Possible values: - `SPOT` - `ON_DEMAND` - `SPOT_WITH_FALLBACK` Default: `SPOT_WITH_FALLBACK` - `zone_id` (string, optional) Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the Databricks deployment. For example, "us-west-2a" is not a valid zone id if the Databricks deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. The list of available zones as well as the default value can be found by using the `List Zones` method. - `instance_profile_arn` (string, optional) Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the Databricks environment by an account administrator. This feature may only be available to certain customer plans. - `spot_bid_price_percent` (int32, optional) The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new `r3.xlarge` spot instance, then the bid price is half of the price of on-demand `r3.xlarge` instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand `r3.xlarge` instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000. Default: `100` - `ebs_volume_type` (string, optional) The type of EBS volumes that will be launched with this cluster. Possible values: - `GENERAL_PURPOSE_SSD` - `THROUGHPUT_OPTIMIZED_HDD` - `ebs_volume_count` (int32, optional) The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail. These EBS volumes will be mounted at `/ebs0`, `/ebs1`, and etc. Instance store volumes will be mounted at `/local_disk0`, `/local_disk1`, and etc. If EBS volumes are attached, Databricks will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, Databricks will configure Spark to use instance store volumes. Please note that if EBS volumes are specified, then the Spark configuration `spark.local.dir` will be overridden. Default: `0` - `ebs_volume_size` (int32, optional) The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096. - `ebs_volume_iops` (int32, optional) If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used. - `ebs_volume_throughput` (int32, optional) If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used. - `azure_attributes` (object, optional) Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used. - `log_analytics_info` (object, optional) Defines values necessary to configure and run Azure Log Analytics agent - `log_analytics_workspace_id` (string, optional) - `log_analytics_primary_key` (string, optional) - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `1` - `availability` (string, optional) Availability type used for all subsequent nodes past the `first_on_demand` ones. Note: If `first_on_demand` is zero, this availability type will be used for the entire cluster. Possible values: - `SPOT_AZURE` - `ON_DEMAND_AZURE` - `SPOT_WITH_FALLBACK_AZURE` Default: `ON_DEMAND_AZURE` - `spot_bid_max_price` (double, optional) The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1. Default: `-1` - `capacity_reservation_group` (string, optional) The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation. Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions: 1. Microsoft.Compute/capacityReservationGroups/read 2. Microsoft.Compute/capacityReservationGroups/deploy/action 3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read 4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action Format: `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}` - `gcp_attributes` (object, optional) Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used. - `use_preemptible_executors` (boolean, optional, Deprecated) This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead. Default: `false` - `google_service_account` (string, optional) If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the Databricks environment by an account administrator. - `boot_disk_size` (int32, optional) Boot disk size in GB - `availability` (string, optional) This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable. Possible values: - `PREEMPTIBLE_GCP` - `ON_DEMAND_GCP` - `PREEMPTIBLE_WITH_FALLBACK_GCP` Default: `ON_DEMAND_GCP` - `zone_id` (string, optional) Identifier for the availability zone in which the cluster resides. This can be one of the following: - "HA" => High availability, spread nodes across availability zones for a Databricks deployment region [default]. - "AUTO" => Databricks picks an availability zone to schedule the cluster on. - A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones. Default: `HA` - `local_ssd_count` (int32, optional) If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to [GCP documentation](https://cloud.google.com/compute/docs/disks/local-ssd#choose_number_local_ssds) for the supported number of local SSDs for each instance type. - `first_on_demand` (int32, optional) The first `first_on_demand` nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, `first_on_demand` nodes will be placed on on-demand instances and the remainder will be placed on `availability` instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster. Default: `1` - `node_type_id` (string, optional) This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the [clusters/listNodeTypes](https://docs.databricks.com/api/workspace/clusters/listnodetypes) API call. - `driver_node_type_id` (string, optional) The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as `node_type_id` defined above. This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence. - `worker_node_type_flexibility` (object, optional) Flexible node type configuration for worker nodes. - `alternate_node_type_ids` (array of string, optional) A list of node type IDs to use as fallbacks when the primary node type is unavailable. - `aws_context_id` (string, optional) The AWS Context ID for EC2 Fleet. When set (non-empty), the value is passed to AWS CreateFleet API to create the EC2 Fleet. - `driver_node_type_flexibility` (object, optional) Flexible node type configuration for the driver node. - `alternate_node_type_ids` (array of string, optional) A list of node type IDs to use as fallbacks when the primary node type is unavailable. - `aws_context_id` (string, optional) The AWS Context ID for EC2 Fleet. When set (non-empty), the value is passed to AWS CreateFleet API to create the EC2 Fleet. - `ssh_public_keys` (array of string, optional) SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name `ubuntu` on port `2200`. Up to 10 keys can be specified. - `custom_tags` (object, optional) Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to `default_tags`. Notes: - Currently, Databricks allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags - `cluster_log_conf` (object, optional) The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every `5 mins`. The destination of driver logs is `$destination/$clusterId/driver`, while the destination of executor logs is `$destination/$clusterId/executor`. - `dbfs` (object, optional) destination needs to be provided. e.g. `{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `volumes` (object, optional) destination needs to be provided, e.g. `{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `spark_env_vars` (object, optional) An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., `export X='Y'`) while launching the driver and workers. In order to specify an additional set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` - `autotermination_minutes` (int32, optional) Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination. - `enable_elastic_disk` (boolean, optional) Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. **AWS:** Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details. - `init_scripts` (array of object, optional) The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If `cluster_log_conf` is specified, init script logs are sent to `//init_scripts`. - `dbfs` (object, optional, Deprecated) destination needs to be provided. e.g. `{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `file` (object, optional) destination needs to be provided, e.g. `{ "file": { "destination": "file:/my/local/file.sh" } }` - `destination` (string, required) local file destination, e.g. `file:/my/local/file.sh` - `gcs` (object, optional) destination needs to be provided, e.g. `{ "gcs": { "destination": "gs://my-bucket/file.sh" } }` - `destination` (string, required) GCS destination/URI, e.g. `gs://my-bucket/some-prefix` - `abfss` (object, optional) destination needs to be provided, e.g. `abfss://@.dfs.core.windows.net/` - `destination` (string, required) abfss destination, e.g. `abfss://@.dfs.core.windows.net/`. - `workspace` (object, optional) destination needs to be provided, e.g. `{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }` - `destination` (string, required) wsfs destination, e.g. `workspace:/cluster-init-scripts/setup-datadog.sh` - `volumes` (object, optional) destination needs to be provided. e.g. `{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `docker_image` (object, optional) Custom docker image BYOC - `url` (string, optional) URL of the docker image. - `basic_auth` (object, optional) Basic auth with username and password - `username` (string, optional) Name of the user - `password` (string, optional) Password of the user - `instance_pool_id` (string, optional) The optional ID of the instance pool to which the cluster belongs. - `single_user_name` (string, optional) Single user name if data_security_mode is `SINGLE_USER` - `policy_id` (string, optional) The ID of the cluster policy used to create the cluster if applicable. - `enable_local_disk_encryption` (boolean, optional) Whether to enable LUKS on cluster VMs' local disks - `driver_instance_pool_id` (string, optional) The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned. - `workload_type` (object, optional) - `clients` (object, required) defined what type of clients can use the cluster. E.g. Notebooks, Jobs - `notebooks` (boolean, optional) With notebooks set, this cluster can be used for notebooks Default: `true` - `jobs` (boolean, optional) With jobs set, the cluster can be used for jobs Default: `true` - `data_security_mode` (string, optional) Possible values: - `NONE` - `SINGLE_USER` - `USER_ISOLATION` - `LEGACY_TABLE_ACL` - `LEGACY_PASSTHROUGH` - `LEGACY_SINGLE_USER` - `LEGACY_SINGLE_USER_STANDARD` - `DATA_SECURITY_MODE_STANDARD` - `DATA_SECURITY_MODE_DEDICATED` - `DATA_SECURITY_MODE_AUTO` - `runtime_engine` (string, optional) Determines the cluster's runtime engine, either standard or Photon. This field is not compatible with legacy `spark_version` values that contain `-photon-`. Remove `-photon-` from the `spark_version` and set `runtime_engine` to `PHOTON`. If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used. Possible values: - `NULL` - `STANDARD` - `PHOTON` - `kind` (string, optional) Possible values: - `COMPUTE_KIND_UNSPECIFIED` - `CLASSIC_PREVIEW` - `use_ml_runtime` (boolean, optional) This field can only be used when `kind = CLASSIC_PREVIEW`. `effective_spark_version` is determined by `spark_version` (DBR release), this field `use_ml_runtime`, and whether `node_type_id` is gpu node or not. - `is_single_node` (boolean, optional) This field can only be used when `kind = CLASSIC_PREVIEW`. When set to true, Databricks will automatically set single node related `custom_tags`, `spark_conf`, and `num_workers` - `remote_disk_throughput` (int32, optional) If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks. - `total_initial_remote_disk_size` (int32, optional) If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP. - `dependency_mode` (string, optional, Beta) Controls dependency configuration for the cluster. Possible values: - `DEPENDENCY_MODE_UNSPECIFIED` - `DEPENDENCY_MODE_ENVIRONMENTS` - `DEPENDENCY_MODE_CLUSTER_LIBRARIES` - `DEPENDENCY_MODE_AUTO` - `previous_cluster_size` (object, optional) The size of the cluster before an edit or resize. - `num_workers` (int32, optional) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are provisioned. - `autoscale` (object, optional) Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later. - `min_workers` (int32, optional) The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation. - `max_workers` (int32, optional) The maximum number of workers to which the cluster can scale up when overloaded. Note that `max_workers` must be strictly greater than `min_workers`. - `cluster_size` (object, optional) The actual cluster size that was set in the cluster creation or edit. - `num_workers` (int32, optional) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are provisioned. - `autoscale` (object, optional) Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later. - `min_workers` (int32, optional) The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation. - `max_workers` (int32, optional) The maximum number of workers to which the cluster can scale up when overloaded. Note that `max_workers` must be strictly greater than `min_workers`. - `cause` (string, optional) The cause of a change in target size. Possible values: - `AUTOSCALE` - `USER_REQUEST` - `AUTORECOVERY` - `REPLACE_BAD_NODES` - `AUTOSCALE_V2` - `DBR_AUTOSCALE` - `reason` (object, optional) A termination reason: * On a TERMINATED event, this is the reason of the termination. * On a RESIZE_COMPLETE event, this indicates the reason that we failed to acquire some nodes. - `code` (string, optional) status code indicating why the cluster was terminated Possible values: - `UNKNOWN` - `USER_REQUEST` - `JOB_FINISHED` - `INACTIVITY` - `CLOUD_PROVIDER_SHUTDOWN` - `COMMUNICATION_LOST` - `CLOUD_PROVIDER_LAUNCH_FAILURE` - `INIT_SCRIPT_FAILURE` - `SPARK_STARTUP_FAILURE` - `INVALID_ARGUMENT` - `UNEXPECTED_LAUNCH_FAILURE` - `INTERNAL_ERROR` - `INSTANCE_UNREACHABLE` - `REQUEST_REJECTED` - `TRIAL_EXPIRED` - `DRIVER_UNREACHABLE` - `SPARK_ERROR` - `DRIVER_UNRESPONSIVE` - `METASTORE_COMPONENT_UNHEALTHY` - `DBFS_COMPONENT_UNHEALTHY` - `EXECUTION_COMPONENT_UNHEALTHY` - `AZURE_RESOURCE_MANAGER_THROTTLING` - `AZURE_RESOURCE_PROVIDER_THROTTLING` - `NETWORK_CONFIGURATION_FAILURE` - `CONTAINER_LAUNCH_FAILURE` - `INSTANCE_POOL_CLUSTER_FAILURE` - `SKIPPED_SLOW_NODES` - `ATTACH_PROJECT_FAILURE` - `UPDATE_INSTANCE_PROFILE_FAILURE` - `DATABASE_CONNECTION_FAILURE` - `REQUEST_THROTTLED` - `SELF_BOOTSTRAP_FAILURE` - `GLOBAL_INIT_SCRIPT_FAILURE` - `SLOW_IMAGE_DOWNLOAD` - `INVALID_SPARK_IMAGE` - `NPIP_TUNNEL_TOKEN_FAILURE` - `HIVE_METASTORE_PROVISIONING_FAILURE` - `AZURE_INVALID_DEPLOYMENT_TEMPLATE` - `AZURE_UNEXPECTED_DEPLOYMENT_TEMPLATE_FAILURE` - `SUBNET_EXHAUSTED_FAILURE` - `BOOTSTRAP_TIMEOUT` - `STORAGE_DOWNLOAD_FAILURE` - `CONTROL_PLANE_REQUEST_FAILURE` - `BOOTSTRAP_TIMEOUT_CLOUD_PROVIDER_EXCEPTION` - `AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE` - `DOCKER_IMAGE_PULL_FAILURE` - `AZURE_VNET_CONFIGURATION_FAILURE` - `NPIP_TUNNEL_SETUP_FAILURE` - `AWS_AUTHORIZATION_FAILURE` - `NEPHOS_RESOURCE_MANAGEMENT` - `STS_CLIENT_SETUP_FAILURE` - `SECURITY_DAEMON_REGISTRATION_EXCEPTION` - `AWS_REQUEST_LIMIT_EXCEEDED` - `AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE` - `AWS_UNSUPPORTED_FAILURE` - `AZURE_QUOTA_EXCEEDED_EXCEPTION` - `AZURE_OPERATION_NOT_ALLOWED_EXCEPTION` - `NFS_MOUNT_FAILURE` - `K8S_AUTOSCALING_FAILURE` - `K8S_DBR_CLUSTER_LAUNCH_TIMEOUT` - `SPARK_IMAGE_DOWNLOAD_FAILURE` - `AZURE_VM_EXTENSION_FAILURE` - `WORKSPACE_CANCELLED_ERROR` - `AWS_MAX_SPOT_INSTANCE_COUNT_EXCEEDED_FAILURE` - `TEMPORARILY_UNAVAILABLE` - `WORKER_SETUP_FAILURE` - `IP_EXHAUSTION_FAILURE` - `GCP_QUOTA_EXCEEDED` - `CLOUD_PROVIDER_RESOURCE_STOCKOUT` - `GCP_SERVICE_ACCOUNT_DELETED` - `AZURE_BYOK_KEY_PERMISSION_FAILURE` - `SPOT_INSTANCE_TERMINATION` - `AZURE_EPHEMERAL_DISK_FAILURE` - `ABUSE_DETECTED` - `IMAGE_PULL_PERMISSION_DENIED` - `WORKSPACE_CONFIGURATION_ERROR` - `SECRET_RESOLUTION_ERROR` - `UNSUPPORTED_INSTANCE_TYPE` - `CLOUD_PROVIDER_DISK_SETUP_FAILURE` - `SSH_BOOTSTRAP_FAILURE` - `AWS_INACCESSIBLE_KMS_KEY_FAILURE` - `INIT_CONTAINER_NOT_FINISHED` - `SPARK_IMAGE_DOWNLOAD_THROTTLED` - `SPARK_IMAGE_NOT_FOUND` - `CLUSTER_OPERATION_THROTTLED` - `CLUSTER_OPERATION_TIMEOUT` - `SERVERLESS_LONG_RUNNING_TERMINATED` - `AZURE_PACKED_DEPLOYMENT_PARTIAL_FAILURE` - `INVALID_WORKER_IMAGE_FAILURE` - `WORKSPACE_UPDATE` - `INVALID_AWS_PARAMETER` - `DRIVER_OUT_OF_DISK` - `DRIVER_OUT_OF_MEMORY` - `DRIVER_LAUNCH_TIMEOUT` - `DRIVER_UNEXPECTED_FAILURE` - `UNEXPECTED_POD_RECREATION` - `GCP_INACCESSIBLE_KMS_KEY_FAILURE` - `GCP_KMS_KEY_PERMISSION_DENIED` - `DRIVER_EVICTION` - `USER_INITIATED_VM_TERMINATION` - `GCP_IAM_TIMEOUT` - `AWS_RESOURCE_QUOTA_EXCEEDED` - `CLOUD_ACCOUNT_SETUP_FAILURE` - `AWS_INVALID_KEY_PAIR` - `DRIVER_POD_CREATION_FAILURE` - `MAINTENANCE_MODE` - `INTERNAL_CAPACITY_FAILURE` - `EXECUTOR_POD_UNSCHEDULED` - `STORAGE_DOWNLOAD_FAILURE_SLOW` - `STORAGE_DOWNLOAD_FAILURE_THROTTLED` - `DYNAMIC_SPARK_CONF_SIZE_EXCEEDED` - `AWS_INSTANCE_PROFILE_UPDATE_FAILURE` - `INSTANCE_POOL_NOT_FOUND` - `INSTANCE_POOL_MAX_CAPACITY_REACHED` - `AWS_INVALID_KMS_KEY_STATE` - `GCP_INSUFFICIENT_CAPACITY` - `GCP_API_RATE_QUOTA_EXCEEDED` - `GCP_RESOURCE_QUOTA_EXCEEDED` - `GCP_IP_SPACE_EXHAUSTED` - `GCP_SERVICE_ACCOUNT_ACCESS_DENIED` - `GCP_SERVICE_ACCOUNT_NOT_FOUND` - `GCP_FORBIDDEN` - `GCP_NOT_FOUND` - `RESOURCE_USAGE_BLOCKED` - `DATA_ACCESS_CONFIG_CHANGED` - `ACCESS_TOKEN_FAILURE` - `INVALID_INSTANCE_PLACEMENT_PROTOCOL` - `BUDGET_POLICY_RESOLUTION_FAILURE` - `IN_PENALTY_BOX` - `DISASTER_RECOVERY_REPLICATION` - `BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG` - `INSTANCE_UNREACHABLE_DUE_TO_MISCONFIG` - `STORAGE_DOWNLOAD_FAILURE_DUE_TO_MISCONFIG` - `CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG` - `CLOUD_PROVIDER_LAUNCH_FAILURE_DUE_TO_MISCONFIG` - `GCP_SUBNET_NOT_READY` - `CLOUD_OPERATION_CANCELLED` - `CLOUD_PROVIDER_INSTANCE_NOT_LAUNCHED` - `GCP_TRUSTED_IMAGE_PROJECTS_VIOLATED` - `BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED` - `EOS_SPARK_IMAGE` - `NO_MATCHED_K8S` - `LAZY_ALLOCATION_TIMEOUT` - `DRIVER_NODE_UNREACHABLE` - `SECRET_CREATION_FAILURE` - `POD_SCHEDULING_FAILURE` - `POD_ASSIGNMENT_FAILURE` - `ALLOCATION_TIMEOUT` - `ALLOCATION_TIMEOUT_NO_UNALLOCATED_CLUSTERS` - `ALLOCATION_TIMEOUT_NO_MATCHED_CLUSTERS` - `ALLOCATION_TIMEOUT_NO_READY_CLUSTERS` - `ALLOCATION_TIMEOUT_NO_WARMED_UP_CLUSTERS` - `ALLOCATION_TIMEOUT_NODE_DAEMON_NOT_READY` - `ALLOCATION_TIMEOUT_NO_HEALTHY_CLUSTERS` - `NETVISOR_SETUP_TIMEOUT` - `NO_MATCHED_K8S_TESTING_TAG` - `CLOUD_PROVIDER_RESOURCE_STOCKOUT_DUE_TO_MISCONFIG` - `GKE_BASED_CLUSTER_TERMINATION` - `ALLOCATION_TIMEOUT_NO_HEALTHY_AND_WARMED_UP_CLUSTERS` - `DOCKER_INVALID_OS_EXCEPTION` - `DOCKER_CONTAINER_CREATION_EXCEPTION` - `DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION` - `DNS_RESOLUTION_ERROR` - `GCP_DENIED_BY_ORG_POLICY` - `SECRET_PERMISSION_DENIED` - `NETWORK_CHECK_NIC_FAILURE` - `NETWORK_CHECK_DNS_SERVER_FAILURE` - `NETWORK_CHECK_STORAGE_FAILURE` - `NETWORK_CHECK_METADATA_ENDPOINT_FAILURE` - `NETWORK_CHECK_CONTROL_PLANE_FAILURE` - `NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE` - `DRIVER_UNHEALTHY` - `USAGE_POLICY_ENTITLEMENT_DENIED` - `K8S_ACTIVE_POD_QUOTA_EXCEEDED` - `CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED` - `NETWORK_CHECK_NIC_FAILURE_DUE_TO_MISCONFIG` - `NETWORK_CHECK_DNS_SERVER_FAILURE_DUE_TO_MISCONFIG` - `NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG` - `NETWORK_CHECK_METADATA_ENDPOINT_FAILURE_DUE_TO_MISCONFIG` - `NETWORK_CHECK_CONTROL_PLANE_FAILURE_DUE_TO_MISCONFIG` - `NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE_DUE_TO_MISCONFIG` - `DBR_IMAGE_RESOLUTION_FAILURE` - `CONTROL_PLANE_CONNECTION_FAILURE` - `CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG` - `RATE_LIMITED` - `MTLS_PORT_CONNECTIVITY_FAILURE` - `HIVEMETASTORE_CONNECTIVITY_FAILURE` - `type` (string, optional) type of the termination Possible values: - `SUCCESS` - `CLIENT_ERROR` - `SERVICE_FAULT` - `CLOUD_FAILURE` - `parameters` (object, optional) list of parameters that provide additional information about why the cluster was terminated - `user` (string, optional) The user that caused the event to occur. (Empty if it was done by the control plane.) - `previous_disk_size` (int64, optional) Previous disk size in bytes - `disk_size` (int64, optional) Current disk size in bytes - `free_space` (int64, optional) - `instance_id` (string, optional) Instance Id where the event originated from - `did_not_expand_reason` (string, optional) - `driver_state_message` (string, optional) More details about the change in driver's state - `job_run_name` (string, optional) Unique identifier of the specific job run associated with this cluster event * For clusters created for jobs, this will be the same as the cluster name - `init_scripts` (object, optional) List of global and cluster init scripts associated with this cluster event. - `reported_for_node` (string, optional) The private ip of the node we are reporting init script execution details for (we will select the execution details from only one node rather than reporting the execution details from every node to keep these event details small) This should only be defined for the INIT_SCRIPTS_FINISHED event - `global` (array of object, optional) The global init scripts associated with this cluster event. - `dbfs` (object, optional, Deprecated) destination needs to be provided. e.g. `{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `file` (object, optional) destination needs to be provided, e.g. `{ "file": { "destination": "file:/my/local/file.sh" } }` - `destination` (string, required) local file destination, e.g. `file:/my/local/file.sh` - `gcs` (object, optional) destination needs to be provided, e.g. `{ "gcs": { "destination": "gs://my-bucket/file.sh" } }` - `destination` (string, required) GCS destination/URI, e.g. `gs://my-bucket/some-prefix` - `abfss` (object, optional) destination needs to be provided, e.g. `abfss://@.dfs.core.windows.net/` - `destination` (string, required) abfss destination, e.g. `abfss://@.dfs.core.windows.net/`. - `workspace` (object, optional) destination needs to be provided, e.g. `{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }` - `destination` (string, required) wsfs destination, e.g. `workspace:/cluster-init-scripts/setup-datadog.sh` - `volumes` (object, optional) destination needs to be provided. e.g. `{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `status` (string, optional) The current status of the script Possible values: - `UNKNOWN` - `FAILED_FETCH` - `FAILED_EXECUTION` - `NOT_EXECUTED` - `SKIPPED` - `SUCCEEDED` - `FUSE_MOUNT_FAILED` - `execution_duration_seconds` (int32, optional) The number duration of the script execution in seconds - `error_message` (string, optional) Additional details regarding errors (such as a file not found message if the status is FAILED_FETCH). This field should only be used to provide *additional* information to the status field, not duplicate it. - `stderr` (string, optional) The stderr output from the init script execution. Only populated when init scripts debug is enabled and script execution fails. - `cluster` (array of object, optional) The cluster scoped init scripts associated with this cluster event. - `dbfs` (object, optional, Deprecated) destination needs to be provided. e.g. `{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }` - `destination` (string, required) dbfs destination, e.g. `dbfs:/my/path` - `s3` (object, optional) destination and either the region or endpoint need to be provided. e.g. `{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }` Cluster iam role is used to access s3, please make sure the cluster iam role in `instance_profile_arn` has permission to write data to the s3 destination. - `destination` (string, required) S3 destination, e.g. `s3://my-bucket/some-prefix` Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs. - `region` (string, optional) S3 region, e.g. `us-west-2`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `endpoint` (string, optional) S3 endpoint, e.g. `https://s3-us-west-2.amazonaws.com`. Either region or endpoint needs to be set. If both are set, endpoint will be used. - `enable_encryption` (boolean, optional) (Optional) Flag to enable server side encryption, `false` by default. - `encryption_type` (string, optional) (Optional) The encryption type, it could be `sse-s3` or `sse-kms`. It will be used only when encryption is enabled and the default type is `sse-s3`. - `kms_key` (string, optional) (Optional) Kms key which will be used if encryption is enabled and encryption type is set to `sse-kms`. - `canned_acl` (string, optional) (Optional) Set canned access control list for the logs, e.g. `bucket-owner-full-control`. If `canned_cal` is set, please make sure the cluster iam role has `s3:PutObjectAcl` permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set `bucket-owner-full-control` to make bucket owner able to read the logs. - `file` (object, optional) destination needs to be provided, e.g. `{ "file": { "destination": "file:/my/local/file.sh" } }` - `destination` (string, required) local file destination, e.g. `file:/my/local/file.sh` - `gcs` (object, optional) destination needs to be provided, e.g. `{ "gcs": { "destination": "gs://my-bucket/file.sh" } }` - `destination` (string, required) GCS destination/URI, e.g. `gs://my-bucket/some-prefix` - `abfss` (object, optional) destination needs to be provided, e.g. `abfss://@.dfs.core.windows.net/` - `destination` (string, required) abfss destination, e.g. `abfss://@.dfs.core.windows.net/`. - `workspace` (object, optional) destination needs to be provided, e.g. `{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }` - `destination` (string, required) wsfs destination, e.g. `workspace:/cluster-init-scripts/setup-datadog.sh` - `volumes` (object, optional) destination needs to be provided. e.g. `{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }` - `destination` (string, required) UC Volumes destination, e.g. `/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` or `dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh` - `status` (string, optional) The current status of the script Possible values: - `UNKNOWN` - `FAILED_FETCH` - `FAILED_EXECUTION` - `NOT_EXECUTED` - `SKIPPED` - `SUCCEEDED` - `FUSE_MOUNT_FAILED` - `execution_duration_seconds` (int32, optional) The number duration of the script execution in seconds - `error_message` (string, optional) Additional details regarding errors (such as a file not found message if the status is FAILED_FETCH). This field should only be used to provide *additional* information to the status field, not duplicate it. - `stderr` (string, optional) The stderr output from the init script execution. Only populated when init scripts debug is enabled and script execution fails. - `enable_termination_for_node_blocklisted` (boolean, optional) Whether or not a blocklisted node should be terminated. For ClusterEventType NODE_BLACKLISTED. - `current_num_vcpus` (int32, optional) The current number of vCPUs in the cluster. - `target_num_vcpus` (int32, optional) The targeted number of vCPUs in the cluster. - `data_plane_event_details` (object, optional) - `event_type` (string, optional) Possible values: - `NODE_BLACKLISTED` - `NODE_EXCLUDED_DECOMMISSIONED` - `timestamp` (int64, optional) - `host_id` (string, optional) - `executor_failures` (int32, optional) - `next_page` (object, optional, Deprecated) Deprecated: use next_page_token or prev_page_token instead. The parameters required to retrieve the next page of events. Omitted if there are no more events to read. - `cluster_id` (string, required) The ID of the cluster to retrieve events about. Example: `1234-567890-reef123` - `start_time` (int64, optional) The start time in epoch milliseconds. If empty, returns events starting from the beginning of time. Example: `1617238800000` - `end_time` (int64, optional) The end time in epoch milliseconds. If empty, returns events up to the current time. Example: `1619485200000` - `order` (string, optional) The order to list events in; either "ASC" or "DESC". Defaults to "DESC". Possible values: - `DESC` - `ASC` Default: `DESC` - `event_types` (array of string, optional) An optional set of event types to filter on. If empty, all event types are returned. Possible values: - `CREATING` - `STARTING` - `RESTARTING` - `TERMINATING` - `EDITED` - `RUNNING` - `RESIZING` - `NODES_LOST` - `UPSIZE_COMPLETED` - `INIT_SCRIPTS_STARTED` - `INIT_SCRIPTS_FINISHED` - `DID_NOT_EXPAND_DISK` - `EXPANDED_DISK` - `FAILED_TO_EXPAND_DISK` - `DRIVER_HEALTHY` - `DRIVER_NOT_RESPONDING` - `DRIVER_UNAVAILABLE` - `SPARK_EXCEPTION` - `METASTORE_DOWN` - `DBFS_DOWN` - `AUTOSCALING_STATS_REPORT` - `NODE_BLACKLISTED` - `PINNED` - `UNPINNED` - `NODE_EXCLUDED_DECOMMISSIONED` - `ADD_NODES_FAILED` - `AUTOSCALING_BACKOFF` - `AUTOMATIC_CLUSTER_UPDATE` - `AUTOSCALING_FAILED` - `CLUSTER_MIGRATED` - `DECOMMISSION_STARTED` - `DECOMMISSION_ENDED` - `DEFERRED_POLICY_ENFORCEMENT_SCHEDULED` - `DEFERRED_POLICY_ENFORCEMENT_FAILED` - `UC_VOLUME_MISCONFIGURED` - `offset` (int64, optional, Deprecated) Deprecated: use page_token in combination with page_size instead. The offset in the result set. Defaults to 0 (no offset). When an offset is specified and the results are requested in descending order, the end_time field is required. Example: `5` - `limit` (int64, optional, Deprecated) Deprecated: use page_token in combination with page_size instead. The maximum number of events to include in a page of events. Defaults to 50, and maximum allowed value is 500. Default: `50` Constraints: `<= 500` - `page_token` (string, optional) Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of events respectively. If page_token is empty, the first page is returned. - `page_size` (int32, optional) The maximum number of events to include in a page of events. The server may further constrain the maximum number of results returned in a single page. If the page_size is empty or 0, the server will decide the number of results to be returned. The field has to be in the range [0,500]. If the value is outside the range, the server enforces 0 or 500. Default: `50` - `total_count` (int64, optional, Deprecated) Deprecated: Returns 0 when request uses page_token. Will start returning zero when request uses offset/limit soon. The total number of events filtered by the start_time, end_time, and event_types. - `next_page_token` (string, optional) This field represents the pagination token to retrieve the next page of results. If the value is "", it means no further results for the request. - `prev_page_token` (string, optional) This field represents the pagination token to retrieve the previous page of results. If the value is "", it means no further results for the request. ## Response ```json { "events": [ { "cluster_id": "1202-211320-brick1", "event_details": { "username": "admin" }, "event_type": "RESTARTING", "timestamp": 1509572145487 }, { "cluster_id": "1202-211320-brick1", "event_details": { "termination_reason": { "code": "USER_REQUEST", "parameters": { "username": "admin" } } }, "event_type": "TERMINATING", "timestamp": 1509505807923 } ], "next_page": { "cluster_id": "1202-211320-brick1", "end_time": 1509572145487, "offset": 50, "order": "DESC" }, "total_count": 303 } ```