Customized Notes For Google Cloud Professional Cloud Architect
Customized Notes For Google Cloud Professional Cloud Architect
Lifecycle Policy
Long term storage of data
When should you use signed URL's
application fails with HTTP 408, 429, and 5xx response codes while sending req
gRPC framework
Cloud Storage bucket lock and retention policy
For POSIX file system use Filestore
Persistent Disks?
Network
Dedicated Interconnect
Cloud VPN
How to configure the flow of traffic in Network?
Creating a private connection between On prem and GCE VM
While Migrating from Onprem to GC, how to keep the resources accesible using I
VPN Gateway and VPN Tunnel
Cloud VPN
solution for global load balancing based on the URL path being requested
improve the cache hit ratio using CDN
How bandwith applies to interconnects and VPN
Loadbalancers ans it's supported traffic??
VPC Service Controls
How to communicate a resource in one VPC to others?
Private Google Access enables internal access to Google APIs only
Cloud NAT can be used to access to third-party services on the internet
Cloud CDN and where is it used?
What is Cloud VPN Tunnel?
Global Load Balancing
VPC Service Controls
Interconnect
Dedicated Interconnect
Partner Interconnect
Peering
Direct Peering
Carrier Peering
VPC
Subnet
VPC Routing - Static routes
Cloud Router - Dynamic Routes
Border Gateway Protocol
Shared VPC - Centralised control with shared VPC
VPC Peering
Cloud VPN
VPN Gateway
Bastion Host
NAT Gateway (Network Address Translation) - protect your network with NAT
VPN Tunnel
Shared VPC
Cloud Router
Firewall Rules
Internal IP (Private)
External IP (Public)- limiting public IP's is the best
Private Service Connect
Compute
HTTPS Load Balancing
App Engine
App Engine
Google StackDriver logging agent
VM's
Persistent Disks size increase with minimal downtime
SSD Persistent Disks
Resilience Testing?
LB in GKE
Helm
Preemptile VM's
MIG Templates and Versioning
deploy your custom Java application to Google App Engine
GKE applications and updating them with zero downtime
Monitor and maximize machine utilisation of application
GKE cluster to automatically add or remove nodes based on CPU load.
Preemptile VM's
GKE StatefulSets
For Migs
Regional Persistent Disks
Databases
Time series data from sensors
BigTable
During peak loads, RDB crashes and replica never promoted to master
Datastore indexes are missing and throwing an error in app engine
App engine standard or flex when connecting onprem db over private
For a HA in Cloud SQL
BigQuery Time partioned table expiration
Point in time data recovery for Cloud SQL
Sharding
Read Replicas
How to migrate MySQL db to Cloud SQL
Cloud SQL Proxy?
BigTable Rowkey
Each Database and their Usecase
Operations
Blue-green Deployment
Canary Deployment
Stackdriver
Cloud Deployment Manager
GCP Resource Hierarchy
Resilience testing
Analytics
BigQuery
BigQuery
Dataflow
Want to move Hadoop workloads to cloud with minimal changes in code
Security
Cloud Identity
PCI data to be used for analytics
Google Secrets Manager
PCI DSS-compliant environment
Effective IAM Policy
Google groups in IAM?
Cloud Armor
Identity and Aware Proxy
Migration
Transfer Appliance
Storage Transfer Service
gsutil
Anthos Service Mesh
JSON Formatting and push it back to bucket using gsutil
GCS
you might not want to require your users to have a Google account in order to access Cloud Storage" "Signed URLs contain au
Implement retry logic using a truncated exponential backoff strategy
gRPC is a high-performance RPC framework that can be used in place of HTTP
enables you to transfer large amounts of data between networks(with low latency), which can be more cost effective than pu
Not good option when the data transfer is huge and with frequent update. Supports only upto 3 GBPS
Refer to target filtering. We can use routing to route the flow, however if FW option is giben, then it's the recommended way
Can use Cloud VPN and Interconnects, but if you need speed then go for Interconnects
Primary
You wantand secondarya ranges
to establish can't
Compute conflict
Engine with on-premises
application in a singleIPVPC
ranges if you
across twohave connected
regions. your VPCmust
The application network to anotherov
communicate n
How should you deploy the VPN?
Cloud VPN is a secure and reliable solution for establishing a connection between your on-premises network and GCP. It uses
You can use URL Maps to configure the HTTPS load balancer to route traffic based on the URL path being requested.
Cloud CDN uses the complete request URL to build the cache key. For performance and scalability, it’s important to optimize
With dedicated interconnect the existing ISP becomes irrelevant. If you were trying to use VPN the existing internet connectio
Resources within a perimeter are accessed only from clients within authorized VPC networks using Private Google Access with
VPC1 instance to VPC2 instance with NIC1 and Connect VPC1 instance to VPC3 instance with NIC2. And update firewall rules t
The Cloud CDN is the best practice for the content caching.
It is responsible for distributing HTTP and HTTPS traffic to backend in order to manage and distribute load in a scalable way
VPC Service Controls provides an extra layer of security defense for Google Cloud services that is independent of Identity and
To setup highspeed direct connection to Google's network for faster data migration in an optimal hybrid environment. Provid
Dedicated Interconnect provides a direct physical connection between your on-premises network and Google's network. 10 G
Partner Interconnect provides connectivity between your on-premises and VPC networks through a supported service provid
You can use Direct Peering to directly connect (peer) with Google Cloud at a Google edge location. You can use Carrier Peerin
Direct Peering enables you to establish a direct peering connection between your business network and Google's edge netwo
Carrier Peering enables you to access Google applications, such as Google Workspace, by using a service provider to obtain en
IP Segment in a VPC(a block of IP Addresses for a region). We have Automatic and Custom (use this to avoid overlapping of IP
GoogleRouter
Cloud Cloud is
routes
a fullydefine the paths
distributed and that network
managed traffic
Google takesservice
Cloud from athat
VMuses
instance to other
the Border destinations.
Gateway These
Protocol destinations
(BGP) to adverti
It programs custom dynamic routes based on the BGP advertisements that it receives from a peer.
A protocol that helps route traffic to subnets
Which allows organisations to connect resources from multiple projects to a common VPC network so that they can commun
Network Peering connects two Virtual Private Cloud (VPC) networks so that resources in each network can communicate with
Google Compute Engine Virtual Private Network (VPN) lets you connect your existing network to your Compute Engine netwo
VPN gateways provide secure connectivity between multiple sites, such as on-premises data centers, Google Cloud Virtual Pr
An external
Routes endpoint
traffic and letsthat allows
multiple the in
VM's clients SSH reach
a subnet from the
thepublic internet
internet using aand canpublic
single still keeps your apps from public facing. Mo
IP address.
Cloud NAT, Google Cloud's managed network address translation service, enables you to provision your application instances
FW rules allow you to isolate your internal network and instances from unwanted access. Allows us to monitor inbound and o
Ephermel and Static IP address, Static IP's are regional resources
Ephermel and Static IP address, can create as external IP's in the network tab. talking through the internet.
resilience test is not about load, is about terminate resources and service not affected. The best for resilience in to introduce
Kubernetes Engine offers integrated support for two types of Cloud Load Balancing (Ingress and External Network Load Balan
Deploy application bundles using dynamic templates
When to use, start, stop -> preemptible vms can be stopped and started anytime with statup/shutdown script = metadata
Use GKE with machine config, build a docker image with dependencies and tags. Create a new deployment with ifnotpresent
Configure a HorizontalPodAutoscaler with a target CPU usage. Enable the Cluster Autoscaler from the GCP Console
For not time-critical applications
To ensure that a workload in Kubernetes has a consistent set of hostnames even after pod scaling and relaunches, you should
To minimize the startup time for new VMs in the instance group, you should create a custom VM image with all of the OS pac
Regional persistent disk is a storage option that provides synchronous replication of data between two zones in a region
Regional persistent disks can be a good building block to use when you implement HA services in Compute Engine
Binary authorization is a feature of Google Kubernetes Engine that allows you to enforce policies on the images that are deplo
NoSQL DB to use for unstructured data
IoT data, handle large volumes of NoSQL data with low latency
Implement routinely scheduled failovers of your databases
Point gcloud datastore create-indexes to your configuration file
App Engine flex is better suitable for accessing On-prem DB while connecting through Cloud VPN, as we have more control ov
Create a failover replica instance in the same region, but in a different zone. Read replicas are used to read only purpose
You can control partition expiration using the time_partitioning_expiration flag in the bq command-line
Enable automatic recovery and binary logging
Sharding is a technique for distributing data across multiple servers to improve performance and scalability.
Read replicas are copies of a database that can be used to offload read traffic from the primary database. While read replicas
The RowKey is used to sort data within a Cloud Bigtable cluster. If the keys are not evenly spread across the alphabet, it can r
Cloud SQL, Spanner, Alloy DB, BigTable, DataStore, Firestore, Cloud MemoryStore
Canary but not on the QA environments. With Canary you usually test in production
to analyze the errors, troubleshooting...
only supports automation of Google Cloud resources
For moving the existing users to cloud, with minimal disruption and added security, use SAML 2.0 authentication instead of G
Use tokenizer service
store the credentials securely
Even though all GCP services are fully complaint, we need to architect and develop the solution as compliant. GCP provides to
the union of the policy set at that resource and the policy inherited from its parent.
urces from VPC networks that span hybrid environments with Private Google Access on-premises extensions.
internet to provide direct access to CDNs and ISPs, reducing latency and improving overall network performance.
ypted because the VPN connections traverse the internet. Each VPN gateway can support multiple connections.
Failover replica is deprecated, In a disaster recovery scenario, you can promote a replica to convert it to a primary. High availa
When a partition expires, data in the partition is deleted but the partitioned table is not dropped even if the table is empty.
should review your RowKey strategy and ensure that keys are evenly spread across the alphabet.
Anthos Service Mesh’s robust tracing, monitoring, and logging features give you deep insights into how your services are perf
rt it to a primary. High availability can be achieved by serving traffic from replicas.
even if the table is empty.
o how your services are performing, how that performance affects other processes, and any issues that might exist.
Getting Started with Google Cloud Certified Professional Cloud Architect
Google Cloud Certified Professional Cloud Architect
Introduction to Cloud and GCP - Google Cloud Platform
Creating GCP - Google Cloud Platform - Account V2
When to use static and ephermel IP, how to relase the Ip addresses
How and when to use, pricing, best practises.
Creating the VM's with startup script can be helpful for pre install and load the softwares
Instance template can be used to create multiple VM's by also configuring the start up script inside to pre install and load the
By creating a Instance template through the custom image which already has all the configs including the startup script
By SSH into /var/www/html.file, checking FW traffic
Dashboards,
Prereq activities,
(Project, and recommendations
billing account, enable api), SSH into it, configure FW rules, Sole tenant node(dedicated hardware for your lic
VM Manager(OS patch, inventory and config management)
Creating group
Managing a groupof VM's andashow
of VMs do you
a single loadMIG(identical
entity. balance betweenVMs multiple VM's
created by template, features include auto-healing, scaling, an
UnManaged
Create IG (different
a Template (Region,configs grouped together,
Auto Scaling(min and maxnoinstances,
features like MIG)for auto-scaling-cpu utilization, Load balancer utiliza
metrics
Auto Healing, and Health Checks
Explore thethe
Specifying created MIG and it's
new template, configurations
specifying how you want the update to be done(when should the update be done - proactive or
How should the update happen - Max Surge or Max unavailable), similary Rolling Restart/Replace
Distributes load across instance in single or multiple regions. Features - Health check, AutoScaling, LB with single anycast IP. E
NetworkLB,
HTTP(S) Layer(IP),
TCP LB,Transport Layer(TCP,
UDP LB. For TLS, UDP),
HTTP, configure Application Layer(HTTP,
(Backend-service HTTPS,
for incoming SMTP,
traffic, HostSFTP, FTP) rules - determines how y
and Path
Front end - IP(This IP is the frontend IP of your client's requests), PORT, and Protocol.
Same as above, but in depth and creating the load balancer
Demonstration on or
External(complex) how it happens in GCP,
Internal(simple) and
Traffic architectural
and in that whatexplanation
kind of traffic (UDP, TCP, HTTP, HTTPS...). Internal - if traffic is U
the traffic is HTTP or S, then Internal HTTP(S) LB. External - If HTTP/S(advanced traffic management then regional or Global d
Diff types of Load Balancers, type of traffic it allows, proxy or passthrough(how the request is transferred to the backend), de
How LB's can manage the Load inside MIG's and also how it redirects traffic between the multiple Regional MIG's
Review of LB concepts, Flexible architecture with LB (Multi-Regional Microservice, Multiple Microservices, Microservice versio
Build Resiliency, Increasing Availability and scalability, improve performance, improve security and low cost implementations
Are the applications available when the users need them? 4 9's - 4.5 mins down time in month, 5 9's - 25 secs downtime
Multiple regional instance groups for each microservice, and distribute load using global HTTPS load balancing
ability to adapt changes in demand (users, data). Vertical Scaling - increasing the size of an instance. Horizontal Scaling, Instea
Selecting the machine type by exploring the VM Size(Vcpu's), in order to increase the size, we have to stop the vm instance an
Check
For Availability
ML/AI policy
Use cases. whilesupport
Doesnt creatingLive
a VM Instance.
Migration when GPU is attched to VM instance. We can add GPU's to some machin
use GPU's, 1. Opt for GPU family. 2. Opt for Normal family and Add GPU's to it.
Use FW rules to restrict ingress and egress traffic. Use Internal IP's as much as possible. Sole tenant nodes when regulatory ne
Abilityyour
Keep to provide
costs asacceptable behaviour
low as possible. SUD even one or discounts
- Automatic more partsfor
fail. Build resilient
running architecture.
VM instances enableportion
for significant live migration and
of billing hea
mont
Applicable for GKE and GCE instances. Restriction on Machine types - A2 and E2. Also, for App Engine flex and Dataflow insta
Commitment
Pvm's from and
- Short lived you(1 or 3). CUD >80%)
cheaper(upto SUD discounts.
VM's. Max Upto
time 70% based
we can run on machine
a Pvm types
is 24 hrs andand
beGPU's.
stoppedSame restrictions
by GCp anytime.asRecei
SUD
application is fault tolerant, cost sensitive, workload is not immediate. Restrictions (Not always available, no automatic resta
Spot vm's are the latest version of Pvm's. Main difference between them is it doesnt have a min or max runtime requirement
Billed by Sec. Always create Budget Alerts.
Use AutoScaling. Understand SUD's. For predicatble long term workloads, use CUD's. if there are non critical and fault toleran
Most GCP services can be managed with CLI using Gcloud (creating a VM, MIG's, Databases, etc...) - gsutil for GCS, bq for BigQ
gcloud init, and config list
gcloud group(containers, compute, config, dataflow, IAM, functions...) subgroup(instances, images, MIG's, regions, zone...) ac
Cloud Shell is backed by a VM instance, 5 GB of free persistent disk storage in $HOME dir, Prepackaged with auto updates, te
Iaas, Paas,using
Iaas-only Faas,infrastructure
Caas, Saas, Serverless
from the cloud provider and you're responsible for application code and runtime, availability, co
OS upgrades and patches. Paas-Use the platform provided by cloud and you're only responsible for application code, and app
Howfocus
You containers canand
on code be the
usedcloud
to ease the microservices
managed service takearchitecture, container
care of all that offerings
is need to and code
scale your it's orchestration and its
to serve millions advanta
of requests
Eg: AWS Lambda,
Compute Google
Engine - Iaas, Cloud
GKE Functions, Azure
- Caas(Complex FunctionsCloud Run - Caas, Serverless(light weight container loads), App Eng
architecture),
Serverless(Standard goes to Zero if there no req), Cloud Functions - Faas, Serverless,
Earliest managed service of google cloud. Platform as a service. Lesser responsibilities as well as lower flexibility
Simplest way to deploy and scale apps in GC-end to end application management. LB, AS, versioning, traffic splitting
Standard
One - App's per
Application run project(application
in language specificacts
sandboxes-many
as container forrestrictions. Flexible
all the sevices you-want
App instances runs
to create in within
your docker
project. container,
Multiple servic
multiple versions to split traffic between them.
Check the PDF for detailedscale
Automatic-Automatically comparision
based onbetween both Util,
the load(CPU standard
etc..).and flexible
Basic- when is there is ADhoc workloads(when there is a req
Manual - Config specific number of instances to run, it's upto your insticts and exp
Demo on building application in App engine, Editor and commands.
Exploring the App Engine UI, services, traffic, pricing, and creating multiple versions of an application, access both the version
here --no-promote is the command not switiching the traffic to new version while deploying. Als0, set traffic and split traffic b
Traffics splitting can be done in console. Also, creating multiple service from the default service
Managed service
Auto repairs and auto upgrades. Pod and Cluster autoscaling. Cloud Monitoring and Logging config. Supports PD's and Local S
Clusters - Standard(complete ownership), AutoPilot(GKE manages). Demo on creating a Standard Cluster
Connect to the cluster and deploy the microservices/containers in it. Later exposing the service created by deployment. Kube
We can add new Node Pools in the cluster. Workloads(deployments). Services and Ingress
Scaling the Node pools using kubectl and cluster nodes using gcloud - manual scaling
Autoscaling microservice and cluster. Config Map- where you store your application configuration for your microservice. Secr
Making changes to Kubernetes Deployments, and Services using YAML file in GKE Console
Cluster(Master and Worker Node). Master Node(manages the cluster)- API Server, Scheduler, Control Manager, etcd. Worker
Kubelet. Cluster Types(Zonal(Single Zonal and Multi-Zonal) Cluster, Regional, Private, Alpha Clusters
Pod(ephemeral ip) contains
Deployment(manages one or more
new releases with containers. Smallest
zero downtime) deployable
is created unitdeployment
for each in K8S. All containers in aall
and contains podits shares(compute,
versions. Replica
pods are running for the specific version of microservice
Servicesisensures
Ingress that externalapproach
the recommended users aretonot impacted
provide whenaccess
external thereto
are changes
services in internally
a cluster. (killing
Ingressof pods, new
provides Loadversion release,
balancing and
defining rules on the ingress resource
Fully managed contianer Registry(storing images) provided by Google Cloud(Docker Hub)
Understanding the components of Docker file and how to best create the images
Deleting the GKE resources using the delete command in the Cloud Shell. Deleting Service and Deployment using Kubectl, and
Hardware-Cluster(Master node, Worker Node, Node pools). Software-(Pods, Deployments, Services)
Run code in response to the events. serverless, pay for what you use - compute time. 2 Cloud Functions - 1 is the 1st version,
CloudRun and Eventarc
What is Event(Can be triggered from multiple GCP Products like GCS, PubSub...), Trigger, Functions,
Creating the CF by configuring the params and writing small code and later testing the functionality and logs
Container to production in seconds. ANthos - Run kubernetes clusters anywhere(cloud, multicloud, onprem)
Serverless, pay per use, fully integrated end-end dev exp. We can have multiple revisions on one service.
gcloud run deploy Service_Name
Security and Encryption in Cloud. Google-Manages Key, KMS(Customer-managed Key-create the key from KMS, Customer-Su
Data at rest(stored
Generally, on a harddisk).
Data is encrypted Data
at Rest. ButinweMotion(transit)-2 types:the
also need to encrpyt 1. data
In andinout of the
transit. cloud andKey
Symmetric 2. Within the Cloud.
Encryption - Uses Data
sameink
Assymmetric
Create Key Encryption
and manage(can bring-the
2 different keys -on-prem)
keys created public and private
cryptographic keys(both symmentric and asymmetric) and KMS can
GCP Services. KMS provides an API to encrypt, Decrypt and sign data
Demo on KMS, creating a key ring- creating encryption keys inside KMS and how we can attach this KMS key while creating a
Block Storage: 1. Persistant Disks - Network Block Storage (Regional and Zonal replication) -- High Durability. 2. Local SSD's - L
File Storage - FileStore in GC.
What are Local SSD's? How is it compared to PD's. High IOPS, Ephemeral Storage, Locally attached to VM instances. Life time
Provisioned Capacity, Very flexible, independent life cycle from VM instance, Regional twice costlier than zonal
Discussing the key diff in PD and Local SSD's features like - Attachment to VM instance, Lifecycle, I/O Speed, Snapshot suppor
Standard, Balanced, and SSD. Discussing features like Underlying storage(HDD,SSD,SSD), Sequential IOPS, Transactional IOPS,
Snapshot is a backup of PD(regional and multi), can schedule them, shared across the projects, incremental, add similar data
Exploring Disk
additional PD'sCreation new with
to VM along and existing
the BoottoDisk.
VM,How
Creating Snapshots,
to add a new PDsnapshot scheduling.
to a VM that is already running. Steps: 1. Attach the dis
Disk. Resizing the existing PD
Machine Image(Created from VM Instance) is diff from Image(created from the book PD). Machine Image contains everything
Usecase: Clong, Replication, and disk backup of instance
Best scenarios for PD's(improve performance, scheduling, increasing durability, deleting old snapshots....)
Shared Cloud File Storage, Provisioned capacity, supports NFSv3 protocol, usecase: file share, media workloads, content man
Resources created in global, Regional and Zonal
Scenario's for Local SSD, Filestore, Creating a Machine Image, Custom Image
GCS Demo of creating buckets and uploading data into GCS bucket
Serverless: Autoscale and infinite scale. Also known as object storage. Stores large objects using key-value approach. CLI-gsuti
Buckets are globally Unique. Store unlimited number of objects. Max size of one object is 1 TB
Storage Classes help us in optimizing our costs based on our accessing data needs
Standard,atNearline,
Enabled Coldline,
bucket level. and Archival
Prevents Storage
accidental Classes.
deletion GCS has history.
and provides low latency, High availability,
Live version Unlimited
is latest. Older Storage
versions are identified
number)
Object Lifecycle management helps in saving costs by automatically moving files between storage classes. Two kind of actions
adding kms keys while creating a storage bucket. Server side (google managed, CSEK,CMEK)
metadata is nothing but a Key-Value Pair. Each object in GCS bucket can have metadata associated with it. Fixed key metadat
How doTransfer
Online you ensure to or
(gsutil comply
API) -with
less regulatory and compliance
than 1 TB data, req around
and a one time immutable
thing. Storage storage
Transfer in a -GCS
Service Bucket?
if data -- Data
is more thanRetenti
1 TB,
Transfer Appliance - Physical data Transfer, used when data is more than 20 TB, or online transfer is taking more than a week
Avoid use of sensitive data in bucket or object names, store data closer to the region for users. Use Cloud Storage Fuse to ena
IAM in GCP
Is the right user has the right access to the right resource? IAM is all about Authentication, Authorization and providing granu
Member,
Roles Resource, and Actions.set
are permissioms(perform Roles and Permissions
of actions on set of resources). Basic Roles/Primitive Roles(owner, editor, viewer). Predefi
by GCP. Custom roles - created when predefined roles are not sufficient for your purposes.
Hand's on approach on learnng more about roles
Role can have multiple permissions. Members can have multiple roles
Demo on creating users in IAM. Policy Troubleshooting to check whether the role is working or not through an API Call.
Whenever an application on a VM needs access to any resource in GCP, the way we can provide access is through Service Acc
Demo on creating Service account and creating a VM from that service account. Using the VM, creating the buckets in GCS th
Use cases explaining OnPrem to GCS, OnPrem to Google Cloud API's
Cloudhas
Who Storage,
accessPubSub, Service and
to the buckets account as identity
Objects and whatand resource.
level ProjectA
of access Service
they have? account
How accessing
it is diff reosurces
from IAM? Use ACLofwhen
GCS bucket
you n
Individual objects in a bucket. Two types of ACL - Uniform and FineGrained Access.
To allow users limited time access to objects who doesnt have google accounts.
Bucket should be created with same name as domian name, assign all users viewer role and expose it to public.
DataBases in GCP
DB's provide organised and persistent storage for your data. Understand availability, durability, RPO, RTO, consistency, transa
Understanding the concepts of Standby and taking the snapshots for databases
Availability(4
Increasing 9's)-Will I- be
Availability ablemultiple
having to accessstandby's
data nowinand whenzones
multiple I needand
it? Durability(11 9's) - Will
regions. Increasing my data
Durability be available
- Having forcopies
multiple 10, 10
in multiplehow
Measuring zones and regions
quickly we can recover from failure - RTO and RPO. RTO(Recovery Time Objective) - Max acceptable downtim
RPO(Recovery Point Objective) - Max acceptable period of data loss, How much timeperiod we can loss data upto. Achieving
Create Read Replicas to reduce the burden on master database where the usage is just the reads from DB. Scale the Read Rep
How to ensure the data in multiple databases(replicas) is updated simultaneously? - Strong Consistency, Eventual Consistency
Relational DB's(OLTP, OLAP), Document, Graphs, Key-Value
OLTP-Online Transaction processing. Applications where large no.of users make large transactions. Usecases: Banking, Stocks
OLAP-Online Analytics Processing. Applications allowing users to analyze petabytes of data. eg: Datawarehouses, BI Applicatio
NoSQL-Not Only SQL. They tradeoff strong consistency and SQL features to high performance and scalability. Firestore(datast
Retrieving data from memory much faster than doing it from disk. Memorystore in GCP. Usecases: Geospatial applications, ga
oogle Cloud
Cloud SQL and Cloud Spanner: Demo and usage
Fully Managed RDB. Provides Regional Service with HA(99.95), SSD/HDD, 30 TB of data storage. Low cost compared to Spann
DataStore,Scalabale
DS-Highly FireStore,NoSQL
and Cloud BigTable.
DB which autoDemo
scalesand
andunderstanding
partitions dataofasall the DB'sOnly for few TB's of data. Supports transactio
it grows.
Firestore is an upgrade on Datastore for its multi-device access. Offline mode and data synchronization
Choose Native(if new projects) and DataStore mode(if old datastore projects move to firestore). Add data by start collection a
Documentscale
Petabyte storeWide
withcolumn
flexibleNoSQL
schema(storing
DB(HBASEuser profiles, indexes
compatible) forfor
designed objects stored
analytical andinoperational
GCS) data(IOT, Timeseries). Mil
Not Serverless. CBT Command line tool. Each table is sorted Key/Value map
Each table has only one index-rowkey
Recommended for Streaming IOT and Timeseries data. Cross Region/Zone Cluster replication
GC VPC is your own isolated network in Google Cloud. VPC is global resource and contains subnets in one or more regions
Can control
Create all the
seperate traffic for
subnets coming in and public
seperating out of resources
the VPC. Best
fromPractise
privateisresources
to createwithin
all your resources
VPC. Also forwith VPC
distributing resources in
associated with a specific region.
There is a Default VPC, but we can also create VPC's(2 options): 1. AutoMode VPC(Default VPC is Automode)- Subnets are aito
2.CustomMode VPC - Subnets are not automatically created and we have complete control over subnets and their IP ranges.
CIDR block
Demo can help
on creating you and
VPC's express the range
subnets insideof IP addresses
them. Creatingthat
VM'sresources
with newinvpc's
a network cancomunicating
and start have. with each in the same
(ping don't work). Resources in the same VPC can talk to each other, but resources in diff network cannot.
FW rules - Control traffic going in and out of the network.
Use network tags(control incoming and outgoing traffic into a VM using FW rules)
We want
How resources
to connect VPCinnetworks
diff projects talk(with
across internal ip's securely
diff organisations? and (Networks
VPC Peering efficiently)in
tosame,
each other, how toand
diff projects, do that?
acrossShared VPC(c
projects in d
internal Ip address
Cloud Monitoring helps us in monitoring our infrastruture using Tools. Metrics, Create Visualisation and Dashboarding, Config
to organise monitoring info and group all the information from multiple GCP projects and AWS acounts. Install Cloud Monito
Real time log management and analysis tool. Ingest data from any source. Logs Explorer, Log Metrics, Logs Dashboard - Key F
Access Transparency
Manage the logs usingLogs(only for gold level
Log Router(what or above),
to ingest, Cloud
discard and Audit
whereLogs(Admin Activity
to route). Two typesLogs, Data
of log Access_Required(Holds
buckets. Logs, System Event
Ad
_Default Logs(all other logs). Can export your logs to GCS Buckets, BQ datasets. Use LogRouter for creating sinks to export lo
Creating a GCS Bucket and Process it with Cloud Function (on when a new object is uploaded into the bucket)
Logs Explorer(filtering the logs), Log Storage(log buckets), Log Router(contains rules for Log buckets, and creation of Sink)
Distributing tracing system for GCP, collecting latency data from supported GCP services
Capturing the state of a running application directly in GCP Env. No need of code change, aading log statements. Take snapsh
Identifying performance bottlenecks in production - Cloud Profiler, statistical and low-header profiler. conitnuosly gathers CP
Identifying prod problems in real-time
Stackdriver is an oldname for Cloud Monitoring, Logging, Tracing, .......
Org>Folder>Project>Resources.
Mandatory for creating resourcesCreate seperate
in a project. projects
Billing for diff
account canenv, seperate folders
be associated for or
with one each department.
more projects. Can have multiple
Cloud Billing Budget to avoid surprises(alert thresholds - 50,90, and 100%). Billing data can be exported to BQ, and GCS bucke
Principle ofusing
Enterprise LeastGoogle
privilege. Separation
workspace of duties(involve
or Enterprises 2 people
uses an identityfor sensitive
provider of tasks).
its ownConstant
(ADID likeMonitoring of Audit Logs.
SAML for vodafone Use
login), in
with the identity provider using Cloud Identity Platform and enable single sign on.
Google Account (A Person or email I'd), Service Account (An Application account), Google Group(collection of google and serv
to enable
To groups).centralised
Google Workspace Domin(manage
constraints on all resourcesfrom workspace
created if you use the domainan
in a organisation-configure fororg
google cloud)
policy. Need to have an Organisa
Iam focuses on who and org policy focuses on what(disable creation of service accounts). Org Policy overwrites IAM
IAM policy can be set at any level of of the heirarchy(org, level, folder, project, and resource). Policy inheritance from parents
Data Operations and Query Operations. BQ admin has full control. Data roles have access to data and Job roles have access to
Federate Cloud Identity or G Workspace with external identity provider(IDP) such as Active Directory or Azure AD
When to use what?
GCE Linux VM uses ssh-key based authentication(metadata and OS Login). Metadata(individually managing ssh keys). OSLogin
Console commands
Execute - SSH Button. also, aGcloud
before Command - gcloud
vm is terminated, compute
stopped, ssh. Next option
or restarted(perform is customized
cleanup or exportssh
of keys(upload public
logs, applicable forkey
bothtopm
VM's. Very similar to startup script however stored as metadata key value. Won't run if you use hard reset.
Quota errors, boot disk full, check serial port output, disk have valid file system
VM instances can be moved only across the zones with in a region but not across the regions. Done using Gcloud move comm
Synchronoud
In Async comm, communication
create a topicmay
and be
haveat your
fault apps
whenput
thelog
web application
messages goes
on the down,
topic andallthe
thelogging
messages willpicks
service be lost andfor
them cannot
proceb
are not lost even when a subscriber went down.
Reliable,Flow
PubSub Autoscalable,
--> Topic isLow-cost,
created, fully managed are
Subscriptions async messaging
created. service.
Published Supports
sends eventtoingestion
a message and delivery
topic, messages of streaming
individually delive
(subscribers can receive the message either by push or pull), and subscriber sends the acknowledgement and then,
Creating a Topic, multiple subscriptions, multiple messages to publish among the individual subscribers and play around with message
Pubsub lite for zonal replication and low cost. Also the snapshot for subscriptions.
Converting Sync to Async workflows. PubSub alternatives - RabbitMQ, Kafka. Adding Dataflow into the flow for enabling mess
Based on Apache Beam, serverless, and autoscaling. Streaming and Batch usecase. can build pipelines from pre-built templat
Connecting onprem
Connecting resources
toofgcp
on-prem toover
network the resources withintraffic)
internet(public Googleusing
Cloud
IPSec VPN Tunnel and traffic encrypted through Interne
gives low throuput outcome. Two types of Cloud VPN. HA VPN - 99.99% SLA with two external IP's. Classic VPN - 99.9% SLA w
High Speed, HA, lowlatency private connection into GCP from your companies onprem network. Dedicated Interconnect(idea
Ensure resources uses different range of IP addresses. Have a fallback option incase primary connection fails. Direct peering is
Big Queries
Realtime, -- Huge volumes
serverless, scalable,ofrelational
data query- pay for the
databasing amount of
warehouse. data scanned
Importing byofthe
variety query in
sources not for theformates
various data returned. Loadi
including str
DataStudio. Configuring Table Expiration. Can query external datasources without storing the data in BQ. Can be
Partitioning dividing table into multiple segments (Ingestion Time, or by columns(Timestamp, date, datetime, Integer). Cluste accessed us
contents of one or more columns. If partitions are less than 1 GB, then choose Clustering.
Pay for data stored in BQ. Config Default table Expiration for datasets, config expiration time for tables, config partition expira
Batch(free import), Stream(expensive - PubSub, streaming inserts, also from Dataflow and Dataproc), DataTransferService(im
Add insertid
Estimate yourwith eachbefore
queries streaming insert
running to avoid
them. duplicates. Strict
use partitioning streaming Avoid
and clustering. quotasstreaming
in BQ. If you're streaming
whenever millions
it's possible. of rows
Expire da
Storage option. BQ is fast for complex Queries, for simple queries choose BT(narrow-range queries)
Managed Spark and Hadoop Service in GCP. Can have multiple cluster modes(single, standard, HA-3masters)-use regular/pree
loads to Google Cloud, then Dataproc(complex batch processing)
Flow of data
Managing in GCP.
data Different
and getting stages in Data
intelligence out ofLifecycle and
it is what different strive
companies relevant
for.services that areorused
Ingest(stream at each
batch), stage. Durable an
Store(Highly
(Converting data into information). Explore and Visualize(Play with data and Get insights from the data)
Streaming
Cloud - Pubsub,
Storage, CloudBatch(StorageTransferService,
SQL, Cloud Spanner, Cloud Firestore, BQTransferService, Transfer
Cloud Bigtable, Appliance,
BigQuery, Customgsutil), Database Migration fromyo
Databases(marketplace-deply o
Store data in the right format against the right databases
RawData to Actionable
BigQuery(complex Info(cleaning,
analysis on pb's of transforming, anonymization).
data using queries), Dataprep(clean
Vertex AI, Pre-built and preparetext,
ML Models(vision, data), DLP(mask, tokenise
Speechtotext, NL...), Das
DataStudio(Dashboarding,
Best recommended GCP Services and visualization),
for different DataCatalouge
usecases in the(Data Discovery and Metadata
Data Lifecycle(Pubsub, management)
BQ, DataStudio, Dataflow, Dataproc, Da
Batch and Streaming flows with right set of services and implementations - refer doc
IOTCore is a GC Service for managing IoT devices(authorisation, registration, and auth) - They send/recieve messages or telem
Centralised platform for storing data with the combination for DataStorage, Data Management, and Analytics.
Goals(zero downtime, one version live at a time, min costs, test with prod traffic before going live) and best practises of Relea
incremental changes)
Most basic approach. Disadvantages(app is down during new release, rollback needs redeployment(again downtime). It is cos
Canary testing:
Rolling: New version
New version is rolled
rolled out out to a subset
to a percentage of instances
of instances, and and then test
if it works fine,with
thenthe live traffic,
gradually if the
rollout testing
this is successful
new version to the
Blue Green: Creating or replicating a parallel environment with new version and then switch all traffic from old version tofailu
downtime, needs automation and additional setups.Main advantage is that the user impact is very low incase of release new
approach, zero downtime, easy rollback, extra infra needed. Shadow testing: Similar to Bluegreen testing, however after crea
4 Phases
Rehosting(lifet and shift)-take the app as is and deploy it to cloud. Replatforming - Make a few adjustments to suit the cloud.
1. Asses the workload to be migrated. 2 Plan the foundation. 3 Deploy the workloads. 4 Optimize your workloads
Example migrations of MySQL Db, and a containarised application to GC
Cron Job Scheduling service in Google Cloud. Fully Managed enterprise-grade scheduler. Integrates with GAE, PubSub, Cloud
Developing GC applications in your local machine without connecting to GCP- setup local dev env using Cloud emulators. See
Global Domain Naming System. DNS is just a way to manage the mapping from a name to an ipaddress. Managed Zones and
Estimates for 40+ Services. Quick recap before the exam would be great.
Centrally manage multiple kubernetes clusters which are created in AWS, GC, and On-prem
Pre-built API's, AutoML, Vertex AI, Data management(cloud storage, BQ, and BQML)
All needs of API's in GC can be taken care by Apigee
Customer identity and access management platform (CIAM) for you end users of your applications, not for your resources like
Eventarc adheres to the cloud events specification. Event Provider(Direct - who can trigger events -pubsub, CF, GCS,..Indirect
indirect),
Using logs,and Eventand
metrics, Destination(CF, CR, GKE).
traces to measure the Uses Pubsub
internal statetopics
of its on the bg.
system Demo on eventarc
by measuring it's outputs - Observability. Open tele
sdk, tools) to collect and export telemetry - metrics, traces, and logs
What is Service Discovery(Help MS help one another) and Service Directory(a single place to connect, discover, and publish se
Best practises and recommendations from GC to help design the deployments. Focus on designing robust, secure, and scalab
Strategies to fulfill those
data Sec controls, manage auth and authorisation, Compute sec controls, securing the network
Getting Ready
nection from supported partner. Data is not encrypted by interconnects.