Semiar reports Cloud computing
Semiar reports Cloud computing
“CLOUD COMPUTING”
SUBMITTED BY
Priyanka Gehlot
1
Faculty of Information Technology
Lucky Institute of Professional Studies
Jodhpur
CERTIFICATE
Ms. Priya Arora The report has been completed and submitted as part of the
requirements for the award of Bachelor in Computer Applications from
Lucky Institute of Professional Studies.
This report has been carried out during the academic year 2024-2025 and is an
authentic record of the work done by Priyanka Gehlot
Date:
2
Acknowledgment
The success and final outcome of this seminar report required a lot of guidance
and assistance from many people and I am extremely privileged to have got this
all along the completion of the report. All that I have done is only due to such
supervision and assistance and we would not forget to thank them.
I am grateful to the mentor Ms. Priya Arora(Asst. Prof.) for giving guidelines
to make the report successful. The interest and attention which has shown so
graciously lavished upon this work.
I extend my thanks to Dr. Saurabh Khatri (HoD, IT) for his cooperation,
guidance, encouragement, inspiration, support and attention led to complete this
report.
I would like to give sincere thanks to Dr. Manish Kachhawaha (Director) and
Mr. Arjun Singh Sankhala (Principal) for providing cordial environment to
exhibit my abilities to the fullest.
Yours Sincerely,
Priyanka Gehlot
3
Declaration
I hereby declare that this Seminar Report is a record of original work done by
me under the supervision and guidance of Ms. Priya Arora . I further certify that
this report work has not formed the basis for the award of the Degree/Diploma
or similar work to any candidate of any university and no part of this report is
reproduced as it is from any source without seeking permission.
Roll no:
Date:
4
CLOUD COMPUTING
Abstract
The term “cloud computing” is a recent buzzword in the IT world. Behind this
fancy poetic phrase there lies a true picture of the future of computing for both
in technical perspective and social perspective. Though the term “Cloud
Computing” is recent but the idea of centralizing computation and storage in
distributed data centers maintained by third party companies is not new but it
came in way back in 1990s along with distributed computing approaches like
grid computing. Cloud computing is aimed at providing IT as a service to the
cloud users on-demand basis with greater flexibility, availability, reliability and
scalability with utility computing model. This new paradigm of computing has
an immense potential in it to be used in the field of e-governance and in rural
development perspective in developing countries like India.
5
Table of Content
S.no Particular Page No
1. Introduction 7
6 Conclusion 38
7 Reference 39
6
CLOUD COMPUTING
1 Introduction
Cloud computing is a new and fast-growing way of using computers over the internet, but the
idea isn't brand new. Back in 1969, a scientist named L. Kleinrock predicted that one day,
computers would work like utilities (like electricity or water), providing services to homes
and businesses. This idea is now what we call "cloud computing."
In the mid-1990s, a similar concept called grid computing was introduced, which allowed
people to access computing power whenever they needed it. Over time, this evolved into
what we now know as cloud computing. The term "cloud computing" became popular around
2006, especially after Google's CEO Eric Schmidt started using it.
Cloud computing works by connecting many computers (called servers) that are spread out
across different locations. These computers work together to provide services like storage,
software, or platforms those users can access over the internet. A key technology that makes
this possible is virtualization, which allows these servers to act as virtual machines. This
means users can use computing resources without having to worry about the physical
machines behind them.
Cloud services are offered on a pay-as-you-go basis, meaning you only pay for what you use,
similar to paying for electricity or water. These services are often grouped as "XaaS," where
"X" can be Software (SaaS), Platform (PaaS), or Infrastructure (IaaS). This way, users don’t
have to buy or maintain expensive hardware and software—they can simply rent it from
cloud providers.
Cloud computing is becoming very popular because it is flexible, reliable, and can easily
grow (scale) based on demand. While businesses are the main users of cloud computing, it
can also help with social issues. For example, in countries like India, where many people
depend on agriculture, cloud computing could improve farming methods and government
services.
In short, cloud computing saves money, makes things easier for users, and can be used in
many different ways. Its growth is driven by its ability to offer services that are both
7
convenient and cost-effective, without the need for users to manage complex IT infrastructure
themselves.
Cloud computing is growing now-a-days in the interest of technical and business organizations but
this can also be beneficial for solving social issues. In the recent time E-Governance is being
implemented in developing countries to improve efficiency and effectiveness of governance. This
approach can be improved much by using cloud computing instead of traditional ICT. In India,
economy is agriculture based and most of the citizens live in rural areas.
The standard of living, agricultural productivity etc can be enhanced by utilizing cloud computing in a
proper way. Both of these applications of cloud computing have technological as well as social
challenges to overcome. In this report we would try to clarify some of the ideas – Why is cloud
computing a buzzword today? i.e. what are the benefits the provider and the users get using cloud?
Though its idea has come long back in 1990 but what situation made it indispensable today? How is
cloud built? What differentiates it from similar terms like grid computing and utility computing?
What are the different services are provided by the cloud providers? Though cloud computing now-a-
days talks about business enterprises not the non-profit organizations; how can this new paradigm be
used in the services like e-governance and in social development issues of rural India?
8
1. Private Cloud – This type of cloud is maintained within an organization and used solely
for their internal purpose. So the utility model is not a big term in this scenario. Many
companies are moving towards this setting and experts consider this is the 1st step for an
organization to move into cloud. Security, network bandwidth are not critical issues for
private cloud.
2. Public Cloud – In this type an organization rents cloud services from cloud providers
on-demand basis. Services provided to the users using utility computing model.
3. Hybrid Cloud – This type of cloud is composed of multiple internal or external cloud.
This is the scenario when an organization moves to public cloud computing domain from its
internal private cloud.
To know why cloud computing is used let’s first concentrate on who use it. And then we
would discuss what advantages they get using cloud. There are three types of stakeholders
cloud providers, cloud users and the end users [Figure 1]. Cloud providers provide cloud
services to the cloud users. These cloud services are of the form of utility computing i.e. the
cloud users uses these services pay-as-you-go model. The cloud users develop their product
using these services and deliver the product to the end users.
The advantages for using cloud services can be of technical, architectural, business etc [5, 6].
9
1. Cloud Providers’ point of view
(a) Most of the data centers today are under utilized. They are mostly 15% utilized. These
data centers need spare capacity just to cope with the huge spikes that sometimes get in the
server usage. Large companies having those data centers can easily rent those computing
power to other organizations and get profit out of it and also make the resources needed for
running data center (like power) utilized properly.
(b) Companies having large data centers have already deployed the resources and to provide
cloud services they would need very little investment and the cost would be incremental .
(a) Cloud users need not to take care about the hardware and software they use and also they
don’t have to be worried about maintenance. The users are no longer tied to some one
traditional system.
(b) Virtualization technology gives the illusion to the users that they are having all the
resources available
. (c) Cloud users can use the resources on demand basis and pay as much as they use. So the
users can plan well for reducing their usage to minimize their expenditure
. (d) Scalability is one of the major advantages to cloud users. Scalability is provided
dynamically to the users. Users get as much resources as they need. Thus this model perfectly
fits in the management of rare spikes in the demand.
Cloud computing is not a new idea but it is an evolution of some old paradigm of distributed
computing. The advent of the enthusiasm about cloud computing in recent past is due to some
recent technology trend and business models
1. High demand of interactive applications – Applications with real time response and
with capability of providing information either by other users or by nonhuman sensors
gaining more and more popularity today. These are generally attracted to cloud not
only because of high availability but also because these services are generally data
intensive and require analyzing data across different sources
2. Parallel batch processing – Cloud inherently supports batch-processing and
analyzing Tera-bytes of data very efficiently. Programming models like Google’s
10
map-reduce [18] and Yahoo!’s open source counter part Hadoop can be used to do
these hiding operational complexity of parallel processing of hundreds of cloud
computing servers.
3. New trend in business world and scientific community – In recent times the
business enterprises are interested in discovering customer’s needs, buying patterns,
supply chains to take top management decisions. These require analysis of very large
amount of online data. This can be done with the help of cloud very easily. Yahoo!
Homepage is a very good example of such thing. In the homepage they show the
hottest news in the country. And according to the users’ interest they change the ads
and other sections in the page. Other than these many scientific experiments need very
time consuming data processing jobs like LHC (Large Hadron Collider). Those can be
done by cloud
3. Cloud Architecture
The cloud providers actually have the physical data centers to provide virtualized services to
their users through Internet. The cloud providers often provide separation between
application and data. This scenario is shown in the Figure 2. The underlying physical
machines are generally organized in grids and they are usually geographically distributed.
Virtualization plays an important role in the cloud scenario. The data center hosts provide the
physical hardware on which virtual machines resides. User potentially can use any OS
supported by the virtual machines used.
11
Figure 2
Operating systems are designed for specific hardware and software. It results in the lack of
portability of operating system and software from one machine to another machine which
uses different instruction set architecture. The concept of virtual machine solves this problem
by acting as an interface between the hardware and the operating system called as system
VMs [21]. Another category of virtual machine is called process virtual machine which acts
as an abstract layer between the operating system and applications.
Virtualization can be very roughly said to be as software translating the hardware instructions
generated by conventional software to the understandable format for the physical hardware.
Virtualization also includes the mapping of virtual resources like registers and memory to
real hardware resources.
The underlying platform in virtualization is generally referred to as host and the software that
runs in the VM environment is called as the guest. The Figure 3 shows very basics of
virtualization. Here the virtualization layer covers the physical hardware. Operating System
accesses physical hardware through virtualization layer. Applications can issue instruction by
using OS interface as well as directly using virtualizing layer interface. This design enables
the users to use applications not compatible with the operating system. Virtualization enables
the migration of the virtual image from one physical machine to another and this feature is
useful for cloud as by data locality lots of optimization is possible and also this feature is
helpful for taking back up in different locations. This feature also enables the provider to shut
down some of the data center physical machines to reduce power consumption
12
4.1. Types of utility cloud services
Utility computing services provided by the cloud provider can be classified by the type of
the services. These services are typically represented as XaaS where we:
13
dynamic. Thus security is
engineered in fundamental
grid infrastructure.
can replace X by Infrastructure or Platform or Hardware or Software or Desktop or Data etc. There
are three main types of services most widely accepted - Software as a Service, Platform as a Service
and Infrastructure as a Service. These services provide different levels of abstraction and flexibility to
the cloud users.
4.1.1. SaaS (Software as a service) – Delivers a single application through the web
browser to thousands of customers using a multitenant architecture. On the customer side, it
means no upfront investment in servers or software licensing; on the provider side, with just
one application to maintain, cost is low compared to conventional hosting. Under SaaS, the
software publisher (seller) runs and maintains all necessary hardware and software. The
customer of SaaS accesses the applications through Internet. For example Salesforce.com
with yearly revenues of over $300M, offers on-demand Customer Relationship Management
software solutions. This application runs on Salesforce.com’s own infrastructure and
delivered directly to the users over the Internet. Salesforce does not sell perpetual licenses but
it charges a monthly subscription fee starting at $65/user/month [10]. Google docs is also a
very nice example of SaaS where the users can create, edit, delete and share their documents,
spreadsheets or presentations whereas Google have the responsibility to maintain the
software and hardware.
E.g. - Google Apps, Zoho Office
14
4.1.3. IaaS (Infrastructure as a Service) – IaaS service provides the users of the cloud
greater flexibility to lower level than other services. It gives even CPU clocks with OS level
control to the developers
. E.g. - Amazon EC2 and S3.
5.1.1 Scalability: Amazon S3 can handle virtually unlimited amounts of data. Whether
you’re storing a few gigabytes or several petabytes, S3 automatically scales
storage infrastructure to meet your needs without manual intervention.
15
supports SDKs for multiple programming languages, making it easy to integrate
into applications..
5.1.6 Data Management and Analytics: S3 provides tools for data management,
including lifecycle policies that automate data transfer between storage classes.
Users can also run analytics directly on data stored in S3 using services like
Amazon Athena, which allows querying data using standard SQL
16
{Resource}: The resource path that you define in your API.
5.2.2 Resources
Resources in API Gateway represent the various parts of your API, such as collections or
specific data objects. Each resource is identified by a path (e.g., /users, /orders/{orderId}),
and you can define multiple methods (GET, POST, DELETE, etc.) for each resource.
Resource Paths: Resources are defined hierarchically, similar to a folder structure,
and can have child resources. For example, /orders/{orderId}/items could be a
resource within an /orders/{ordered} resource.
Methods: Each resource can have one or more methods associated with it,
corresponding to HTTP methods like GET, POST, DELETE, PUT, etc. Methods
define how the API should handle requests sent to that resource.
Integration: Each method is linked to a backend integration, which could be an AWS
Lambda function, an HTTP endpoint, or another AWS service. This integration
defines how API Gateway should process and route the request
5.2.3 API Stages and Deployment
Stages: Stages represent different versions or environments of your API (e.g., dev,
test, prod). Each stage has its own URL endpoint and can have different settings, such
as caching or logging.
Deployment: Deploying an API in API Gateway means making it available at a
specific stage. Once you deploy an API, you can access it via the stage URL, and it
becomes publicly accessible (depending on your configuration).
5.2.4 Security and Authorization
API Gateway provides several options for securing your APIs, including:
API Keys: You can require clients to include API keys in their requests to identify
and authenticate them.
IAM Permissions: Use AWS Identity and Access Management (IAM) policies to
control access to your API.
Cognito User Pools: Integrate with Amazon Cognito to manage user authentication
and authorization.
Lambda Authorizers: Use custom AWS Lambda functions to control access to your
API by implementing custom authorization logic.
Getting Started with Amazon API Gateway
Here’s a step-by-step guide to getting started with Amazon API Gateway.
Step 1: Creating a REST API
17
1. Log in to the AWS Management Console:
o Navigate to the Amazon API Gateway service.
2. Create a New API:
o Click on "Create API".
o Choose "REST API" and then select "Build".
o Choose an API type: "HTTP API" for lightweight APIs or "REST API" for
fully-featured APIs.
o Enter an API name and description.
o Click "Create API".
Step 2: Define Resources and Methods
1. Create a Resource:
o In the API Gateway console, under your newly created API, click on
"Resources".
o Click "Create Resource" to add a new resource.
o Enter a resource name and path (e.g., /users).
2. Add Methods:
o Select the resource you just created.
o Click "Create Method".
o Choose an HTTP method (e.g., GET, POST).
o Select the integration type. You can choose from options like "Lambda
Function", "HTTP", "Mock", or "AWS Service".
o If you choose "Lambda Function", select your Lambda function to integrate
with this method.
3. Set Up Method Response and Integration Response:
o Configure the method response to define how the API responds to the client.
o Configure the integration response to define how API Gateway should handle
responses from the backend service.
Step 3: Deploy the API
1. Create a Stage:
o After defining your resources and methods, you need to deploy your API to a
stage.
o Click on "Deploy API".
o Choose "Deploy API" from the Actions menu.
o Create a new stage (e.g., dev, prod) and provide a description.
18
2. Access Your API:
o After deployment, you will receive an endpoint URL for the stage.
o You can use this URL to test and access your API.
Step 4: Secure and Monitor Your API
1. Enable Security:
o Implement security measures like requiring API keys, setting up IAM
permissions, or integrating with Amazon Cognito.
2. Monitor API Usage:
o Use Amazon Cloud Watch to monitor API usage and performance metrics like
request count, latency, and error rates.
o Set up alarms to notify you of any unusual activity or performance
degradation.
Step 5: Versioning and API Lifecycle Management
1. Create New Versions:
o As you make updates to your API, you can create new versions or stages to
manage the API lifecycle.
o Deploy new versions to different stages for testing or production use.
2. Documentation and SDK Generation:
o Generate API documentation and SDKs (Software Development Kits) for
client developers to integrate with your API.
o Use the built-in documentation editor to provide details on API usage,
methods, and parameters.
Advanced Features and Use Cases
Caching:
API Gateway provides the option to cache responses at the stage level to improve
performance and reduce backend load. You can configure caching settings such as
time-to-live (TTL) and cache key parameters.
Throttling and Rate Limiting:
You can set up throttling limits to control the number of requests that clients can
make to your API within a certain timeframe. This helps protect your backend
services from being overwhelmed by excessive traffic.
Cross-Origin Resource Sharing (CORS):
Enable CORS on your API to allow cross-domain requests from browsers, which is
essential for web applications that interact with your API from different domains.
19
API Gateway Web Socket APIs:
Use Web Socket APIs for real-time communication applications, such as chat apps or
live data feeds. API Gateway manages Web Socket connections and can trigger
backend services based on Web Socket events.
Integrating with Other AWS Services:
API Gateway can be integrated with other AWS services such as AWS Lambda,
Amazon DynamoDB, Amazon S3, and more to build comprehensive serverless
applications.
20
Message Attributes: Additional metadata that you can include with your messages.
Attributes provide structured information about the message that consumers can use
for filtering or routing.
Message Retention: The period during which SQS retains a message if it’s not
processed and deleted by a consumer. The retention period can be configured from 1
minute to 14 days.
Visibility Timeout: The time period during which a message is invisible to other
consumers after being received by a consumer. If the message is not deleted within
this time, it becomes visible again in the queue.
3. Dead-Letter Queues (DLQ)
A Dead-Letter Queue is a special type of queue used to handle messages that cannot be
processed successfully. When a message fails to be processed after a specified number of
attempts, it is moved to a DLQ for further investigation or reprocessing. This feature helps in
identifying problematic messages and preventing them from blocking the processing of other
messages.
4. Long Polling
Long Polling reduces the cost and latency of retrieving messages from SQS by allowing the
consumer to wait for a message to become available instead of polling the queue
continuously. When a long poll request is made, the request waits until a message is available
or the long polling timeout period is reached.
Amazon SNS
1. Topics
Topics: Amazon SNS topics are logical access points that act as communication
channels for sending messages to multiple subscribers. A topic can support multiple
subscriber endpoints, and each message sent to the topic is delivered to all
subscribers.
Topic Attributes: Metadata about a topic, including settings such as display name,
delivery policy, and access control. You can configure these attributes to manage how
messages are delivered and who can publish to or subscribe to the topic.
2. Subscriptions
Subscription Endpoints: These are the destinations where messages published to an
SNS topic are delivered. Endpoints can include Amazon SQS queues, AWS Lambda
functions, HTTP/HTTPS endpoints, email addresses, and mobile push notifications
(e.g., SMS, Apple Push Notification Service).
21
Filtering Policies: Filtering policies allow subscribers to receive only specific
messages that match certain attributes, reducing the volume of irrelevant messages
that subscribers need to process.
3. Message Formats
Raw Message Delivery: This feature allows SNS to send the exact message payload
to subscribers without any additional SNS-generated metadata. It is useful when
subscribers need to receive the raw message content as-is.
Message Structure: SNS supports structured messages that can include different
formats or content for each type of subscriber endpoint. This allows you to tailor the
message content to suit the capabilities and requirements of different subscriber types.
4. Fan-Out Pattern
The Fan-Out Pattern allows you to distribute messages to multiple endpoints
simultaneously by publishing a single message to an SNS topic that has multiple SQS queues
or other endpoints subscribed to it. This is commonly used for distributing workload to
multiple processing nodes or delivering notifications to multiple systems.
Getting Started with Amazon SQS
Step 1: Create an SQS Queue
1. Log in to the AWS Management Console and navigate to the Amazon SQS service.
2. Create a New Queue:
o Choose between a Standard Queue or a FIFO Queue depending on your
application’s requirements.
o Configure queue settings, such as the queue name, visibility timeout, message
retention period, and whether to enable encryption.
Step 2: Send and Receive Messages
1. Send Messages:
o Use the AWS Management Console, AWS CLI, or SDKs to send messages to
the queue.
o When sending a message, you can optionally include message attributes and
set a delay time for the message.
2. Receive and Process Messages:
o Use a consumer application or service to poll the queue for new messages.
o After receiving a message, process it and then delete it from the queue to
prevent it from being processed again.
Step 3: Implementing Dead-Letter Queues
1. Create a Dead-Letter Queue:
22
o Configure a secondary queue to act as a DLQ for your main queue.
o Set the maximum number of receive attempts for the main queue before a
message is moved to the DLQ.
2. Monitor and Handle Failed Messages:
o Monitor the DLQ for messages that couldn’t be processed successfully.
o Investigate and resolve the issues with these messages or reprocess them as
needed.
Getting Started with Amazon SNS
Step 1: Create an SNS Topic
1. Log in to the AWS Management Console and navigate to the Amazon SNS service.
2. Create a New Topic:
o Choose a topic type (Standard or FIFO) based on your requirements.
o Name the topic and configure topic attributes, such as display name and access
policies.
Step 2: Add Subscriptions
1. Subscribe Endpoints to the Topic:
o Create subscriptions by specifying the endpoint type (e.g., SQS, Lambda,
email) and the endpoint address (e.g., email address, queue ARN).
o Confirm subscriptions if required (e.g., email subscriptions require
confirmation).
2. Configure Filtering Policies:
o Set up filtering policies for each subscription if you want to limit the messages
that are delivered to specific subscribers based on message attributes.
Step 3: Publish Messages to the Topic
1. Publish Messages:
o Use the AWS Management Console, AWS CLI, or SDKs to publish messages
to the SNS topic.
o You can include message attributes to allow for filtering and use structured
messages for different subscriber types.
2. Monitor and Analyze Delivery:
o Monitor message delivery success and failures using CloudWatch metrics and
logs.
o Analyze delivery logs to troubleshoot any issues with message delivery or to
gain insights into the performance of your messaging system.
23
Advanced Use Cases and Integration
1. Integrating SQS and SNS
Fan-Out with SQS and SNS: Use SNS to fan out a message to multiple SQS queues.
This pattern is useful when you need to distribute the same message to multiple
processing nodes or different systems.
Event-Driven Architectures: Use SNS to publish events from your application, and
then have different microservices or applications subscribe to these events through
SQS, Lambda, or HTTP endpoints.
2. Combining with AWS Lambda
Lambda Triggers: Set up Lambda functions to trigger in response to messages
arriving in SQS or SNS. This is particularly useful for real-time processing, data
transformations, and integrating with other AWS services.
Serverless Architectures: Build serverless architectures where SNS and SQS handle
message delivery and queuing, and Lambda functions process these messages without
the need for dedicated servers.
3. Monitoring and Logging
CloudWatch Metrics: Use Amazon CloudWatch to monitor key metrics for your
SQS queues and SNS topics, such as message count, delivery success rate, and
throttling.
Logging: Implement logging to track message flow and diagnose issues within your
messaging system. CloudWatch Logs can be used to capture logs for further analysis.
Conclusion
Amazon SQS and SNS are fundamental services for building scalable, decoupled, and
resilient cloud applications. By mastering the key concepts and following the steps to get
started, you can leverage SQS and SNS to create powerful messaging systems that handle
distributed workloads, coordinate microservices, and improve the reliability of your
applications. Whether you’re using SQS for decoupling components or SNS for broadcasting
notifications, these services are essential for modern cloud-based architectures.
5.3.2: Amazon SQS (Simple Queue Service) and Amazon SNS (Simple
Notification Service)
25
o Decoupling: The producer and consumer are decoupled, allowing them to
operate independently.
Real-World Example:
o Order Processing System: In an e-commerce platform, orders are placed into
a queue and processed one at a time by a backend system. Each order is
processed only once by a single worker, ensuring that orders are handled
sequentially and reliably.
2. Publish-Subscribe Messaging (Pub-Sub)
Definition: A messaging model where messages are sent from a producer (publisher)
to multiple consumers (subscribers) through a topic. Each subscriber receives its own
copy of the message.
How It Works:
o The producer publishes a message to a topic.
o All subscribers to that topic receive a copy of the message.
o Each subscriber processes the message independently.
Characteristics:
o One-to-Many Communication: Each message is delivered to all subscribers
of the topic.
o Flexible Distribution: New subscribers can join or leave without affecting the
producer or other subscribers.
o Decoupling: The publisher and subscribers are decoupled, allowing them to
evolve independently.
Real-World Example:
o News Broadcasting: A news agency publishes news updates to a topic. All
subscribed news outlets receive the updates and can choose how to use them
(e.g., display on a website, broadcast on TV, etc.).
Part 3: Fundamentals of Messaging
1. Components of Messaging Systems
Producer: The component or service that sends messages to the messaging system.
Consumer: The component or service that receives and processes messages from the
messaging system.
Message: The data or information being transmitted between producers and
consumers.
26
Queue: A storage area where messages are held until a consumer retrieves them (used
in point-to-point messaging).
Topic: A distribution mechanism for publishing messages to multiple subscribers
(used in pub-sub messaging).
2. Key Concepts in Messaging
Asynchronous Processing: The producer sends a message and continues with its
work without waiting for the consumer to process the message. The consumer
processes the message whenever it is ready.
Message Durability: Ensuring that messages are not lost in transit. This typically
involves storing messages in a durable way until they are acknowledged by the
consumer.
Message Ordering: The sequence in which messages are processed. In some
systems, messages must be processed in the order they are received (FIFO - First In,
First Out), while in others, order may not matter.
3. Benefits of Messaging Systems
Scalability: Messaging systems allow different parts of an application to scale
independently. For example, if the message production rate increases, you can add
more consumers to handle the load without changing the producer.
Reliability: Messages can be stored and retried if delivery fails, ensuring that
communication between components is reliable even in the event of a failure.
Flexibility: Messaging systems can support multiple communication patterns (e.g.,
one-to-one, one-to-many), making them suitable for a wide range of applications.
4. Challenges and Considerations
Message Duplication: Ensuring that messages are not processed more than once
(impotency).
Latency: Minimizing delays in message delivery and processing.
Error Handling: Defining how to handle messages that cannot be processed
successfully (e.g., moving to a dead-letter queue).
27
Job Scheduling: Tasks like data processing jobs are added to a queue and picked up
by worker nodes for execution, allowing for parallel processing.
Email Sending: Emails generated by an application are queued up and sent out one
by one by an email server, ensuring each email is processed and sent.
2. Publish-Subscribe Messaging (Pub-Sub) Use Cases
Push Notifications: An application publishes notifications to a topic. All subscribed
users receive the notification on their devices.
Stock Market Updates: A stock exchange publishes real-time price updates to a
topic. All financial services subscribed to the topic receive the updates for trading
decisions.
Microservices Event Notification: In a microservices architecture, different services
can subscribe to events like "User Created" to perform their respective functions, such
as sending a welcome email, creating a user profile, or logging the event.
28
Open the floor for questions to ensure everyone understands the fundamentals of
messaging systems.
Encourage students to think about how they might use messaging in their
own projects or future work.
Conclusion
By understanding the fundamentals of messaging, point-to-point, and pub-sub models,
students will be better equipped to design scalable and reliable distributed systems. These
notes provide a foundation for further exploration into specific messaging technologies like
Amazon SQS and SNS
29
Event-driven Model: Lambda functions are designed to be triggered by events,
making it ideal for real-time data processing, web services, and other scenarios where
immediate execution is required.
Trigger Configuration: You configure triggers to define which event sources should
invoke your Lambda function and under what conditions.
Getting Started with AWS Lambda
Creating a Lambda Function
1. Open the AWS Management Console:
o Navigate to the AWS Lambda service.
2. Create a Function:
o Click on "Create function".
o Choose an authoring method: "Author from scratch", "Use a blueprint", or
"Browse server less app repository".
o Provide the function name, runtime (e.g., Python 3.8, Node.js 14.x), and an
execution role.
3. Configure Function:
o Set memory allocation and timeout settings.
o Add environment variables if needed.
4. Write or Upload Code:
o Use the inline code editor to write your function code, or upload a .zip file
containing your code and dependencies.
5. Set Up Event Source:
o Choose an event source that will trigger your function, such as an S3 bucket,
DynamoDB table, or API Gateway.
6. Test the Function:
o Create a test event in the Lambda console to simulate an event and verify your
function works as expected.
30
o Use Amazon Cloud Watch to monitor Lambda function metrics, such as
invocation count, duration, and errors.
o Enable logging to capture function logs and review them in Cloud Watch
Logs.
3. Scaling:
o Lambda automatically scales your function in response to incoming requests.
You can control concurrency with reserved concurrency settings to limit the
number of concurrent executions.
Advanced Features and Use Cases
Security
IAM Roles: Assign execution roles to Lambda functions to grant permissions for
accessing other AWS services.
VPC Integration: Configure Lambda functions to access resources in your VPC,
such as RDS databases or EC2 instances.
Data Processing
Real-time File Processing: Use Lambda to process files as they are uploaded to S3.
Stream Processing: Use Lambda with Kinesis or DynamoDB streams to process data
in real time.
Automation
Infrastructure Automation: Automate the deployment and management of AWS
resources using Lambda and CloudFormation.
Scheduled Tasks: Use CloudWatch Events or EventBridge to schedule Lambda
functions to run at specific intervals.
Interesting Lambda Code Ideas
1. Image Processing:
o Use Lambda to automatically resize and optimize images uploaded to an S3
bucket.
o Example: Generate thumbnails for images uploaded to S3.
2. Real-time Data Processing:
o Process data streams from IoT devices using Lambda and Kinesis.
o Example: Analyze sensor data from IoT devices and store the results in
DynamoDB.
3. Chatbot Backend:
o Implement a backend for a chatbot using Lambda and API Gateway.
31
o Example: Create a serverless chatbot for customer support that integrates with
Slack or Facebook Messenger.
4. Serverless REST API:
o Build a RESTful API using Lambda and API Gateway.
o Example: Create a CRUD API for managing a to-do list application.
5. Automated Security Checks:
o Use Lambda to perform security checks on your AWS resources.
o Example: Scan S3 buckets for publicly accessible objects and notify
administrators.
6. PDF Generation:
o Automatically generate PDFs from data or templates.
o Example: Create invoices in PDF format based on order data and email them
to customers.
7. Log Analysis:
o Analyze and aggregate logs from CloudWatch or other logging services.
o Example: Detect and alert on specific patterns or anomalies in application
logs.
8. Notification System:
o Build a notification system using Lambda and SNS.
o Example: Send SMS or email alerts based on specific events in your
application.
9. Data Synchronization:
o Synchronize data between different AWS services or external systems.
o Example: Sync data from DynamoDB to Elasticsearch for search and
analytics.
10. Sentiment Analysis:
o Perform sentiment analysis on social media feeds or customer reviews.
o Example: Analyze tweets mentioning your brand and categorize them as
positive, negative, or neutral.
o
Conclusion
AWS Lambda is a powerful service that enables you to build scalable and efficient serverless
applications. By understanding the key concepts, such as functions and event sources, and
following the steps to get started, you can leverage Lambda to automate tasks, process data,
32
and build robust backend services. With its flexible integration capabilities and automatic
scaling, AWS Lambda opens up a wide range of possibilities for innovative serverless
solutions.
AWS Lambda Restrictions and Limitations
Function and Deployment Package Limits
1. Deployment Package Size:
o Compressed (.zip) file size: Up to 50 MB.
o Uncompressed size: Up to 250 MB, including all dependencies.
2. Layers:
o Each Lambda function can have up to 5 layers.
o Each layer can be up to 50 MB when compressed.
3. Container Images:
o The container image size can be up to 10 GB.
Memory Allocation:
o Lambda functions can allocate memory between 128 MB and 10,240 MB (10
GB), in 1 MB increments.
o The amount of CPU available to a function scales with the amount of memory
configured.
4. Execution Timeout:
o The maximum execution timeout for a Lambda function is 15 minutes (900
seconds).
o You can set the timeout for your function to any value from 1 second to 15
minutes.
5. Ephemeral Storage (/tmp) Space:
o Each Lambda function has access to 512 MB of ephemeral storage in the /tmp
directory.
o This space is used for temporary storage during function execution and is not
persistent across invocations.
6. Request and Response Payload Size:
o The maximum payload size for synchronous invocation requests (i.e., direct
calls) is 6 MB.
o The maximum payload size for asynchronous invocation requests (i.e., events
from other services) is 256 KB.
7. Concurrency Limits:
33
o By default, each AWS account has a soft limit of 1,000 concurrent executions
across all functions in a region.
o You can request an increase in this limit through the AWS Support Center if
needed.
.
Networking and Environment Limits
8. VPC Configuration:
o When Lambda functions are configured to run inside a Virtual Private Cloud
(VPC), they can experience a cold start latency.
o Ensure that Lambda functions have access to sufficient IP addresses in the
subnets where they are deployed.
9. Environment Variables:
o Each Lambda function can have up to 4 KB of environment variables.
o Environment variables are used to store configuration settings and sensitive
information.
Execution Environment Limits
10. Lambda@Edge:
o Lambda@Edge functions have specific limitations, such as a maximum
deployment package size of 1 MB and a maximum execution timeout of 5
seconds.
11. Language and Runtime Support:
o Lambda supports several programming languages and runtimes, including
Node.js, Python, Java, Go, Ruby, and .NET Core.
o Ensure that the runtime and language version you choose is supported by
Lambda.
Logging and Monitoring Limits
12. Logging:
o The maximum size for a single log event in CloudWatch Logs is 256 KB.
o Logging excessive amounts of data can lead to increased costs and slower
function execution times.
IAM and Security Limits
13. IAM Role:
o Lambda functions require an execution role with permissions to access AWS
resources.
34
o Be mindful of the principle of least privilege when assigning permissions to
the execution role.
Performance and Best Practices
14. Cold Starts:
o Cold starts occur when a new instance of a Lambda function is invoked for the
first time or after being idle. Cold starts can cause latency issues.
o To minimize cold starts, consider strategies like keeping functions warm
through scheduled invocations.
15. Function Timeout:
o Functions should be designed to complete within the timeout limit to avoid
unexpected terminations.
o Break down long-running tasks into smaller, manageable chunks.
16. Dependencies:
o Minimize the size of your deployment package by including only necessary
dependencies.
o Use Lambda layers to share common dependencies across multiple functions.
Conclusion
AWS Lambda is a powerful tool for building scalable and serverless applications, but it's
important to be aware of its limitations and constraints. By understanding these restrictions
and designing your functions accordingly, you can optimize performance, manage resource
usage, and ensure the efficient operation of your serverless applications. Keep these
limitations in mind when planning, deploying, and managing your Lambda functions to make
the most of what AWS Lambda has to offer.
35
use cases such as mobile apps, gaming, IoT, real-time bidding, and other performance-
sensitive applications.
Key Concepts
1. Tables
Table: A table in DynamoDB is a collection of data items that share the same primary
key. Unlike traditional relational databases, DynamoDB tables do not enforce a fixed
schema for attributes beyond the primary key.
Primary Key: Each table must have a primary key, which uniquely identifies each
item in the table. DynamoDB supports two types of primary keys:
o Partition Key: A simple primary key consisting of a single attribute (also
known as the hash key).
o Partition Key and Sort Key: A composite primary key consisting of two
attributes. The partition key is used to distribute data across partitions, while
the sort key determines the order of items with the same partition key.
2. Items and Attributes
Items: An item in Dynamo DB is a single record in a table, similar to a row in a
relational database. Each item is uniquely identified by its primary key.
Attributes: Attributes are the individual data elements associated with an item,
similar to columns in a relational database. Dynamo DB items can have a flexible
schema, meaning different items in the same table can have different attributes.
3. Read and Write Capacity
Provisioned Throughput: Dynamo DB allows you to specify the read and write
capacity units (RCUs and WCUs) for your table, which determines how much data
can be read from or written to the table per second.
o Read Capacity Units (RCUs): One RCU represents one strongly consistent
read per second, or two eventually consistent reads per second, for items up to
4 KB in size.
o Write Capacity Units (WCUs): One WCU represents one write per second
for items up to 1 KB in size.
On-Demand Mode: DynamoDB also offers an on-demand mode, which
automatically adjusts read and write capacity based on the traffic to your table,
eliminating the need to manually manage capacity.
4. Secondary Indexes
Global Secondary Index (GSI): A GSI allows you to query data on non-primary key
attributes. GSIs can have a different partition key and sort key from the base table.
36
Local Secondary Index (LSI): An LSI allows you to query data on non-primary key
attributes while using the same partition key as the base table. LSIs must be created
when the table is created and allow up to five per table.
5. Data Consistency
Eventually Consistent Reads: This type of read achieves higher throughput but may
not reflect the most recent write. It is ideal for applications where real-time accuracy
is not critical.
Strongly Consistent Reads: This type of read returns the most up-to-date data,
ensuring consistency. It comes at the cost of reduced throughput compared to
eventually consistent reads.
6. Transactions
Transactions: DynamoDB supports ACID (Atomic, Consistent, Isolated, Durable)
transactions, allowing you to perform multiple operations on one or more items in a
single, all-or-nothing operation. Transactions are useful for maintaining data integrity
in applications that require multiple updates to be performed atomically.
Getting Started with Amazon DynamoDB
Step 1: Create a DynamoDB Table
1. Log in to the AWS Management Console and navigate to the DynamoDB service.
2. Create a New Table:
o Click "Create table".
o Enter a table name and choose a primary key (partition key, and optionally,
sort key).
o Select the read/write capacity mode (Provisioned or On-Demand).
o Optionally, add secondary indexes if you need to query the table based on
attributes other than the primary key.
o Click "Create table" to provision the table.
Step 2: Insert and Query Data
1. Insert Data:
o Navigate to your table and select the "Items" tab.
o Click "Create item" to add new items to the table.
o Enter the attribute values for the item and click "Save".
2. Query and Scan:
37
o Use the "Query" feature to retrieve items based on the primary key or a
secondary index.
o Use the "Scan" feature to retrieve all items in the table. Note that scans can be
resource-intensive and slow, especially on large tables.
Step 3: Configure Read and Write Capacity
1. Manage Provisioned Capacity:
o If using provisioned mode, you can adjust the read and write capacity units to
match your application’s needs.
o Monitor the table’s performance metrics in CloudWatch and adjust the
capacity settings as necessary.
2. Use Auto Scaling:
o Enable auto-scaling to automatically adjust the provisioned throughput based
on usage patterns, ensuring your table can handle varying workloads without
Conclusion
Cloud computing has revolutionized the IT industry by providing scalable, flexible, and cost-
effective computing resources over the internet. It eliminates the need for organizations to
invest in expensive infrastructure, as cloud services operate on a pay-as-you-go model. Cloud
computing is categorized into Software as a Service (SaaS), Platform as a Service (PaaS),
and Infrastructure as a Service (IaaS), each catering to different levels of user needs.
The benefits of cloud computing include enhanced scalability, cost savings, remote
accessibility, and reliability. Businesses leverage cloud computing for data storage,
application hosting, and large-scale data processing. Public, private, and hybrid cloud models
offer varying degrees of control, security, and resource management, allowing organizations
to choose solutions tailored to their needs.
Cloud computing is not just a technological advancement; it also holds social and economic
significance. In developing countries like India, cloud-based solutions can improve e-
governance, rural development, and agricultural productivity by enabling digital
transformation. However, challenges such as data security, privacy concerns, and
dependency on internet connectivity need to be addressed for broader adoption.
38
Leading cloud service providers like Amazon Web Services (AWS), Google Cloud, and
Microsoft Azure offer robust solutions that power businesses, research, and social initiatives.
The increasing demand for big data analytics, artificial intelligence, and the Internet of
Things (IoT) further emphasizes the importance of cloud computing.
In conclusion, cloud computing is shaping the future of digital services by offering efficient,
cost-effective, and scalable solutions.
References
1. Mell, P., & Grance, T. (2011). The NIST Definition of Cloud Computing. National
Institute of Standards and Technology (NIST), Special Publication 800-145.
2. Buyya, R., Broberg, J., & Goscinski, A. (2010). Cloud Computing: Principles and
Paradigms. Wiley.
3. Armbrust, M., Fox, A., Griffith, R., et al. (2010). A View of Cloud Computing.
Communications of the ACM, 53(4), 50-58.
4. Vaquero, L. M., Rodero-Merino, L., Caceres, J., & Lindner, M. (2009). A Break in the
Clouds: Towards a Cloud Definition. ACM SIGCOMM Computer Communication
Review, 39(1), 50-55.
5. Rimal, B. P., Choi, E., & Lumb, I. (2009). A Taxonomy and Survey of Cloud
Computing Systems. Proceedings of the 5th International Joint Conference on INC,
IMS, and IDC, IEEE.
6. Foster, I., Zhao, Y., Raicu, I., & Lu, S. (2008). Cloud Computing and Grid
Computing: A Comparison. IEEE Grid Computing Environments Workshop.
7. Marinescu, D. C. (2013). Cloud Computing: Theory and Practice. Morgan Kaufmann.
39
8. Amazon Web Services (AWS). AWS Documentation. Retrieved from
https://aws.amazon.com/documentation/
9. Google Cloud Platform (GCP). Google Cloud Documentation. Retrieved from
https://cloud.google.com/docs
10. Microsoft Azure. Azure Cloud Computing Services. Retrieved from
https://azure.microsoft.com/en-us/documentation/
11. Oracle Cloud Infrastructure. Oracle Cloud Documentation. Retrieved from
https://docs.oracle.com/en/cloud/
40