cloud-composer-environment

Cloud Composer

• Introduction of Cloud Composer
• Cloud Composer Blueprint
• Disclaimer
• Prerequisites
• Deployment Steps
• Verification of a successful deployment
• Variables
• Outputs

Introduction of Cloud Composer

Based on the open source Apache Airflow project, the Cloud Composer provides infastructual support for worklows. It supports creating, scheduling, monitoring and managing workflows enviornments across clouds and data centers.

Cloud Composer Blueprint

This blueprint demonstrates how to deploy cloud composer on Google Cloud Platform (GCP). Directed Acryclic Graphs (DAGs) are created as a collection of tasks or workflows in a schedule. The purpose is ensuring the execution of tasks are completed in a particular order at the correct times, this is because each task can perform multiple functions. There are 4 main components: GKE Cluster, Airflow Web Server, Airflow Database, Cloud Storage Bucket. The GKE Cluster is where the DAG's can be triggered to run automatically in the existing workflow sequence, as a response or manually. In the Cloud Composer's core, the environment, the composer components run instances. The Airflow Web Server is a log to monitor progress of the workflows, the Database stores long term details of the flow. Finally the Cloud storage bucket, stores access to files of the flow. In order for there to be a flow in the access of files, Connections provide access to each of the various services, and hooks will provide access with other external services through the connections. Uploaded DAG's are sent to the composer environment where each individual tasks is executed by leveraging connections and hooks to interact with other systems.

Disclaimer

The present GCP Terraform Module in this project is set up and intended to be implemented in a FEDRAMP High environment using the Assured Workloads within the Google Cloud Platform (GCP) organization. For Cloud Composer, the core components must be placed directly into the specific network area. The main project must have access/permission from the Shared VPC owner to build in the network or there will be a failed deployment. For a successful deployment, run time is around 25 minutes and destroying it will take around 8 minutes.

Prerequisites

1. Service Account User role (roles/iam.serviceAccountUser) for deploying user.

Deployment Steps

You should see this README and some terraform files.

1. Run cp terraform.tfvars.sample terraform.tfvars to copy the sample variables to your own tfvars file.

2. Update the variables as necessary in your tfvars file.

3. To provision this example, run the following from within this directory:

terraform init
terraform plan to see the infrastructure plan
terraform apply to apply the infrastructure build
terraform destroy to destroy the built infrastructure

Verification of a successful deployment

Use GCP console to verify if the resources have been created.

Variables

name	description	type	required	default
composer_env_name	Name of the Composer environment.	string	✓
main_project_id	Main project ID.	string	✓
network_name	Full path to VPC.	string	✓
network_project_id	The ID of the landing zone project where the VPC is.	string	✓
subnetwork_name	The name of the existing subnetwork to use within the specified VPC network and region.	string	✓
composer_version	Cloud composer version.	string		"composer-3-airflow-2"
region	Google Cloud Region.	string		"us-east4"
sa_account_id	Service account id.	string		"composer-env-account"
sa_display_name	Service account display name.	string		"Service Account for Composer Environment"
service_agent_version	Composer Service Agent version. This must correspond to Composer version.	string		"roles/composer.ServiceAgentV2Ext"

Outputs

name	description	sensitive
airflow_uri	URI for Airflow.
composer_id	Cloud composer id.