Cloud Storage

Book website: www.hands-on-books-series.com © 2019 Arshdeep Bahga & Vijay Madisetti

Overview

• Cloud Storage types

• Amazon S3
• Amazon EFS
• Amazon EBS
• AWS Storage Gateway

© 2019 Arshdeep Bahga & Vijay Madisetti

Storage Types

• In cloud computing environments,

three types of storage solutions
are offered:
• Block storage
• File storage
• Object storage

© 2019 Arshdeep Bahga & Vijay Madisetti

Block Storage

• Block storage operates at the operating system kernel level, and the data is stored
and organized as an array of unrelated blocks.
• In Block storage, the data is stored without any concept of data format or type.
• Block storage is accessed over the network as a Storage Area Network (SAN) using
protocols such as iSCSI.

© 2019 Arshdeep Bahga & Vijay Madisetti

File Storage

• File storage operates at the operating system user level, and the data is stored as
data blocks which are managed by a file system.
• In file storage, data is managed as a named hierarchy of files and folders.
• Files have meta-data associated with them (such as file name, type, and creation
date).
• File storage is accessed over the network as a Network Attached Storage (NAS)
using protocols such as Network File System (NFS) or Common Internet File
System (CIFS).

© 2019 Arshdeep Bahga & Vijay Madisetti

Object Storage

• Object storage operates at the application level, and the data is stored as objects.
• Each object consists of an object identifier (OID), data, and meta-data.
• Object storage is accessed with protocols such as HTTP using REST APIs.

© 2019 Arshdeep Bahga & Vijay Madisetti

Amazon Simple Storage Service (S3)

• Amazon Simple Storage Service(S3) is an online cloud-based data storage infrastructure

for storing and retrieving any amount of data.
• S3 provides a highly reliable, scalable, fast, fully redundant, and affordable storage
infrastructure.
• S3 is a cloud object storage service.
• Data stored on S3 is organized in the form of buckets.
• S3 console provides simple wizards for creating a new bucket and uploading files.
• You can upload any type of file to S3.
• While uploading a file, you can specify the redundancy and encryption options and access
permissions.
• Data stored on S3 is independent of any server and is accessed over the Internet using
the S3 web interface or console, S3 REST APIs or using the AWS SDKs.

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Buckets

• Data stored on S3 is organized in the form of buckets.

• While creating a bucket, a bucket name is provided.
• The bucket name has to be unique across all AWS accounts, and not just within your AWS
account.
• An S3 bucket is created for a specific region.
• By selecting a region for the bucket, you can control where your data is stored.

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Objects

• Objects are the entities which are stored in Amazon S3.

• Objects have object data and meta-data. Objects can store data in any format, and the data stored is
opaque to S3.
• Objects size can range from 0 bytes to 5TB.
• An object consists of a Key, Version ID, Value, Meta-data, Subresources, and Access Control Information.
• Each object is uniquely identified by a combination of Bucket, Key, and an optional Version ID.
• Key is the name that is assigned to the object.
• You can use versioning to keep multiple versions of the same object in a bucket.
• Version ID allows you to identify a specific version of an object.
• The Value of an object is the actual content which you store. Value is a sequence of bytes.
• The Metadata for an object consists of a set of key-value pairs which provide information regarding the
object

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Bucket Policies and ACLs

• By default, access to all resources in S3, including buckets and objects is private.
• You can manage access to the S3 resources using resource-based policies or user-based
policies.
• Resource-based policies include bucket policies and access control lists (ACLs)).
• User-based policies include AWS Identity and Access Management (IAM) policies
attached to users in an AWS account.
• Access Control Lists (ACLs) allow you to grant permissions such as read or write at the
level of a bucket or object.
• With ACLs, you can give read or write permissions to other AWS accounts or to the public.
• Each bucket and object has an ACL attached to it as a subresource.
• An ACL defines which AWS accounts or groups are granted access and the type of
access such as READ, WRITE, READ_ACP, WRITE_ACP, and FULL_CONTROL.

© 2019 Arshdeep Bahga & Vijay Madisetti

Cross-Origin Resource Sharing (CORS) in S3

• Cross-Origin Resource Sharing (CORS) is a mechanism that allows a client application

running at one origin (domain) to have permission to access selected resources from a
different origin (domain).
• You can enable CORS for an S3 bucket using the Amazon S3 console, or by using the
Amazon S3 REST API and the AWS SDKs.
• To allow cross-origin requests, a CORS configuration is created. A CORS configuration
specifies rules such as the origins which are allowed to access a bucket, the allowed
operations or HTTP methods, and other operation-specific information.

© 2019 Arshdeep Bahga & Vijay Madisetti

Versioning in S3

• S3 allows you to enable versioning for a bucket.

• When versioning is enabled, you can have multiple versions of an object in one bucket.
• Versioning helps in protecting your data from accidental or malicious updates or deletes.
• Each version is identified by a version ID.
• You can retrieve and restore every version of every object stored in a bucket for which
versioning is enabled.
• Versioning is enabled at the bucket level and once enabled, it cannot be disabled, but can
only be suspended.
• A bucket in which versioning is enabled maintains one current and zero or more
noncurrent object versions.

© 2019 Arshdeep Bahga & Vijay Madisetti

Encryption in S3

• You can protect sensitive data stored in S3 using encryption.

• You can either use Server-Side Encryption (SSE) or Client-Side Encryption for protecting
data at rest in S3.
• With server-side encryption, S3 encrypts the data at the object level before saving it on
disks in its data centers and decrypts it when you download the objects.
• S3 provides three options for SSE depending on how you choose to manage the
encryption keys:
• SSE-S3 (Server-Side Encryption with Amazon S3-Managed Keys)
• SSE-KMS (Server-Side Encryption with AWS KMS-Managed Keys)
• SSE-C (Server-Side Encryption with Customer-Provided Keys)

© 2019 Arshdeep Bahga & Vijay Madisetti

Static Website Hosting in S3

• Amazon S3 buckets can be used for hosting static websites that don’t require a full web
server.
• A static website includes only static HTML web pages, and static resources such as
images, stylesheets (CSS files), and Javascript files.
• By hosting a static website on S3, you can leverage the scalability, availability, durability,
and security offered by S3.
• When you enable static website hosting for a bucket you get an S3 website URL like
<bucket-name>.s3-website.<AWS-region>.amazonaws.com

© 2019 Arshdeep Bahga & Vijay Madisetti

Transfer Acceleration in S3

• You can enable Transfer Acceleration for a bucket to enable fast, easy, and secure
transfers of files over long distances between your client and the S3 bucket.
• Transfer Acceleration feature in S3 uses the globally distributed edge locations of Amazon
CloudFront to route data from an edge location to S3 over an optimized path.

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Durability and Availability

• A durable storage system ensures that the data stored is not lost due to corruption or
degradation.
• An available storage system is one that continues to remain operational and can deliver
data when requested.
• S3 is designed for 99.999999999% (eleven nines) of durability and 99.99% availability.
• S3 provides such high durability of data by storing data redundantly on multiple systems
in multiple facilities within a region.

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Consistency

• S3 provides read-after-write consistency for PUTS of new objects.

• Whereas for PUTS to existing objects and for object DELETES, S3 offers eventual
consistency.
• Eventual consistency for overwrite PUTS means that if you update an object with a PUT
request, and then read the object with a GET request, you can either get the old data or
updated data.
• Similarly, eventual consistency for DELETES means that if you delete an object and then
read the object with a GET request, you can still read the deleted object. However,
updates to a single key are atomic.

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Storage Classes

• S3 offers various storage classes which are designed for different use cases and differ in
the durability and availability:
• Standard: Standard storage class is designed for frequently accessed data and offers high durability
(99.999999999%), high availability (99.99%), low latency, and high throughput.
• Standard - Infrequently Accessed: Standard - Infrequently Accessed (IA) storage class offers the high
durability (99.999999999%) and availability (99.9%) like the Standard storage class, but it is designed for
long-lived and infrequently accessed data.
• Intelligent Tiering: Intelligent Tiering storage class offers high durability (99.999999999%) and availability
(99.9%) and is designed for long-lived data with changing or unknown access patterns.
• One Zone - Infrequently Accessed: One Zone - Infrequently Accessed (IA) storage class offers high
durability (99.999999999%) and a slightly lower availability (99.5%).
• Glacier: Glacier storage class offers the same durability and availability as the Standard storage class but
at extremely low-cost. Glacier is designed for long-term data archiving with retrieval times ranging from
minutes to hours.
• Reduced Redundancy: Reduced Redundancy storage class offers slightly lower durability (99.99%) and
high availability (99.99%).

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Object Lifecycle Management

• S3 allows you to define lifecycle rules for a group of objects to ensure that the objects are
stored cost-effectively throughout their lifecycle.
• A lifecycle configuration can be attached to an S3 bucket such that it applies to all objects
in the bucket or to a group of objects specified by a prefix.
• There are two types of actions that can be defined in a lifecycle configuration:
• transition action
• expiration action

© 2019 Arshdeep Bahga & Vijay Madisetti

S3 Cross-Region Replication

• Cross-region replication allows you to copy objects from a source bucket in one AWS
region to a destination bucket in another region, automatically and asynchronously.
• Cross-region replication is enabled at the bucket-level by providing the destination bucket
where you want S3 to replicate the objects and an AWS IAM role that Amazon S3 can
assume to replicate objects on your behalf.
• To enable cross-region replication, you must have versioning enabled for the source and
destination buckets.

© 2019 Arshdeep Bahga & Vijay Madisetti

Amazon Elastic File System (EFS)

• Amazon Elastic File System (EFS) provides a highly scalable, available, and durable file storage
that can be used with EC2 instances.
• The storage capacity in an EFS file system is elastic and grows or shrinks as you add or delete
files. EFS supports the Network File System protocols (NFSv4.1 and NFSv4.0).
• EFS file systems are distributed across several storage servers, which enables the file system to
grow elastically and support parallel access from multiple EC2 instances.
• EFS is designed for use cases such as Big Data and Analytics workloads, Media Processing
workflows, Content Management, and Web Serving.
• EFS can provide high throughput coupled with read-after-write consistency, low-latency file
operations, and shared file access, which is required for such use cases.

© 2019 Arshdeep Bahga & Vijay Madisetti

EFS Performance Modes

• EFS provides two performance modes: (1) General Purpose and (2) Max I/O.
• The General Purpose performance mode is suitable for applications requiring low latency such as
web serving, content management, and file serving.
• The Max I/O performance mode, is optimized for applications where a large number of EC2
instances are accessing the file system.

© 2019 Arshdeep Bahga & Vijay Madisetti

EFS Throughput Modes

• EFS provides two throughput modes: (1) Bursting throughput and (2) Provisioned throughput.
• When Bursting throughput mode is used for an EFS file system, the throughput scales as a file
system grows. Bursting throughput mode is useful for file-based workloads, which require low levels
of throughput for most of the time and high levels of throughput occasionally.
• Provisioned throughput mode is useful for applications requiring high levels of throughput than
those allowed by the Bursting throughput mode.

© 2019 Arshdeep Bahga & Vijay Madisetti

Amazon Elastic Block Store (EBS)

• Amazon Elastic Block Store (EBS) provides block level storage volumes for use with EC2
instances.
• EBS is designed for high availability and durability, and the EBS volumes are automatically
replicated within an availability zone.
• You can attach multiple EBS volumes to an EC2 instance; however, a volume can be attached to
only one instance at a time.

© 2019 Arshdeep Bahga & Vijay Madisetti

EBS Volumes

• An EBS Volume is a block-level storage device that can be attached to an EC2 instance. The
benefit of using an EBS volume with an EC2 instance is that it can persist independently from the
life of the instance. A volume attached to an EC2 instance can be used like any other physical hard
drive.
• EBS provides various types of volumes that differ in performance characteristics and cost:
• Magnetic (standard): Magnetic volumes have low performance characteristics.
• General Purpose SSD (gp2): General Purpose SSD volumes offer cost-effective storage that balances price
and performance for a wide variety of workloads.
• Provisioned IOPS SSD (io1): Provisioned IOPS SSD volumes are the highest-performance volumes designed
for mission-critical low-latency or high-throughput workloads.
• Cold HDD (sc1): Cold HDD volumes are designed for less frequently accessed workloads.
• Throughput Optimized HDD (st1): Throughput Optimized HDD volumes are designed for frequently
accessed, throughput-intensive workloads.

© 2019 Arshdeep Bahga & Vijay Madisetti

AWS Storage Gateway

• AWS Storage Gateway is a service connecting an on-premises software appliance with cloud-based
storage to provide seamless and secure integration between an organization’s on-premises IT
environment and AWS’s storage infrastructure.
• You can run Storage Gateway either on-premises as a VM appliance, or as a hardware appliance,
or in AWS as an Amazon EC2 instance.
• AWS Storage Gateway supports three types of storage solutions:
• File Gateway: A file gateway allows you to store files as objects in Amazon S3, with a local cache for low-
latency access to your most recently used data.
• Volume Gateway: A volume gateway provides cloud-backed storage volumes that you can mount as
Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers.
• Tape Gateway: Tape gateway allows you to back up your data to Amazon S3 and archive in Amazon
Glacier by leveraging your existing tape-based backup software infrastructure.

© 2019 Arshdeep Bahga & Vijay Madisetti