Amazon S3 Storage

Contents
AMAZON S3 STORAGE ....................................................................................................................... 1
NOTE: THESE INSTRUCTIONS ARE FOR WINDOWS USERS ERROR! BOOKMARK NOT DEFINED.
1 S3 MANAGEMENT ........................................................................................................................ 3
1.1 Overview of Amazon S3................................................................................................................... 3
1.2 Advantages to Amazon S3 .............................................................................................................. 3
1.3 Amazon S3 Concepts ....................................................................................................................... 3
1.3.1 Buckets ..................................................................................................................................................... 3
1.3.2 Objects ...................................................................................................................................................... 3
1.3.3 Folders ...................................................................................................................................................... 4
1.3.4 Keys........................................................................................................................................................... 4
1.3.5 Regions ..................................................................................................................................................... 4
1.4 Accessing Amgen AWS- S3 files .......................................................................................................... 5
2 INTRODUCTION TO THE S3 CONSOLE .................................................................................... 13
2.1 S3 Browser for easy file navigation ......................................................................................................13

© 2014 ZS Associates
Page - 2
1 S3 Management

1.1 Overview of Amazon S3

Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.

The AWS Management Console makes it easy to store and retrieve any amount of data, at any time, from
anywhere on the web. It gives any user access to the same highly scalable, reliable, fast, inexpensive data
storage infrastructure that Amazon uses to run its own global network of web sites.

1.2 Advantages to Amazon S3

Amazon S3 is intentionally built with a minimal feature set that focuses on simplicity and robustness.
Following are some of advantages of the Amazon S3 service:

• Unlimited storage – There is no limit to the amount of data you can store on Amazon S3

• Standard interface – Amazon S3 uses standards based REST and SOAP interfaces designed to work
with any Internet-development toolkit

• Scalable – Amazon S3 can scale in terms of storage, request rate, and users to support an unlimited
number of web-scale applications

• Reliability – Store data with up to 99.999999999% durability, with 99.99% availability

• Inexpensive – Amazon S3 is built from inexpensive commodity hardware components

1.3 Amazon S3 Concepts

This section describes key concepts and terminology you need to understand to use Amazon S3 effectively.
They are presented in the order you will most like encounter them.

1.3.1 Buckets

A bucket is a container for objects stored in Amazon S3. Every object is contained in a bucket. For example, if
the object named photos/puppy.jpg is stored in the johnsmith bucket, then it is addressable using the
URL http://johnsmith.s3.amazonaws.com/photos/puppy.jpg

Buckets serve several purposes: they organize the Amazon S3 namespace at the highest level, they identify
the account responsible for storage and data transfer charges, they play a role in access control, and they
serve as the unit of aggregation for usage reporting.

You can configure buckets so that they are created in a specific Region. For more information, see Regions.

1.3.2 Objects

Objects are the fundamental entities stored in Amazon S3. When using the console, you can think of them as
being files. Objects consist of data and metadata. The data portion is opaque to Amazon S3. The metadata is
a set of name-value pairs that describe the object. These include some default metadata such as the date last

© 2014 ZS Associates
Page - 3
modified, and standard HTTP metadata such as Content-Type. The developer can also specify custom
metadata at the time the object is stored.

An object is uniquely identified within a bucket by a key (name).

1.3.3 Folders

Folders are available in the AWS Management Console, but are not part of the core Amazon S3 API. You use
folders to group objects in a bucket.

When you create a folder in the AWS Management Console, Amazon S3 creates a zero-byte object with a
forward slash (/) at the end of the object name in your bucket. Amazon S3 interprets the forward slash as a
delimiter when performing list operations. For example, if you create a new folder in the AWS Management
Console called logs, Amazon S3 creates an object called logs/. If you upload an object called history.txt to
the logs folder using the AWS Management Console, the full key name for this object is logs/history.txt.

For more information about how Amazon S3 treats keys, go to Amazon Simple Storage Service Developer
Guide.

1.3.4 Keys

A key is like a file name; it is the unique identifier for an object within a bucket. Every object in a bucket has
exactly one key. Because the combination of a bucket, key, and version ID uniquely identify each object,
Amazon S3 can be thought of as a basic data map between "bucket + key + version" and the object itself.
Every object in Amazon S3 can be uniquely addressed through the combination of the web service endpoint,
bucket name, key, and optionally, a version. For example, in the URL http://doc.s3.amazonaws.com/2006-03-
01/AmazonS3.wsdl, "doc" is the name of the bucket and "2006-03-01/AmazonS3.wsdl" is the key.

1.3.5 Regions

You can choose the AWS Region where Amazon S3 will store the buckets you create. You might choose a
Region to optimize latency, minimize costs, or address regulatory requirements. Amazon S3 Data Consistency
Model

Updates to a single key are atomic. For example, if you PUT to an existing key, a subsequent read might
return the old data or the updated data, but it will never write corrupted or partial data.

Amazon S3 achieves high availability by replicating data across multiple servers within Amazon's data centers.
After a success code is returned, your data is safely stored; however, information about the changes might
not immediately replicate across Amazon S3, and so you might observe the following behaviors:

• A process writes a new object to Amazon S3 and immediately attempts to read it. Until the change is
fully propagated, Amazon S3 might report "key does not exist."

• A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the
change is fully propagated, the object might not appear in the list.

• A process replaces an existing object and immediately attempts to read it. Until the change is fully
propagated, Amazon S3 might return the prior data.

• A process deletes an existing object and immediately attempts to read it. Until the deletion is fully
propagated, Amazon S3 might return the deleted data.

© 2014 ZS Associates
Page - 4
 • A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is
fully propagated, Amazon S3 might list the deleted object.

The US Standard Region provides eventual consistency for all requests. All other regions provide read-after-
write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.

Note
Amazon S3 does not currently support object locking. If two puts are simultaneously made to the same key,
the put with the latest time stamp wins. If this is an issue, you will need to build an object-locking mechanism
into your application.
Updates are key-based; there is no way to make atomic updates across keys. For example, you cannot make
the update of one key dependent on the update of another key unless you design this functionality into your
application.

The following table describes the characteristics of eventually consistent read and consistent read.

Eventually Consistent Read Consistent Read

Stale reads possible No stale reads

Lowest read latency Potential higher read latency

Highest read throughput Potential lower read throughput

For more information about the Amazon S3 Data Consistency Model see the Amazon Simple Storage Service
Developer Guide.

Open the following web browser link for general AWS access: http://console.aws.amazon.com/

1.4 Accessing Amgen AWS- S3 files

1. Log onto the Amgen –Amazon secured portal

For Amgen specific AWS, open the following link in web browser:
https://amgen.signin.aws.amazon.com/console/

Enter the following login information:

a. Account: amgen
b. Username: smahanti
c. Password: radarpoc2014

© 2014 ZS Associates
Page - 5
 2. Select the S3 service

3. Create a Bucket

Before you can upload data into Amazon S3, you must create a bucket to store the data in. Buckets
have configuration properties, including their geographical region, who has access to the objects in the
bucket, and other metadata, such as the storage class of the objects in the bucket.

The console enables you to use folders, which you can store objects in. Folders, like objects, must
reside in a bucket. For more information about using folders, see Working With Folders.

Use the following procedure to create a bucket.

Note

© 2014 ZS Associates
Page - 6
 You are not charged for creating a bucket; you are only charged for storing objects in the bucket and
for transferring objects out of the bucket.

To create a bucket

1. Sign into the AWS Management Console and open the Amazon S3 console
at https://console.aws.amazon.com/s3.

2. Click Create Bucket.

3. In the Create Bucket dialog box, in the Bucket Name box, type a name for your bucket.

The name that you choose must be unique across all existing bucket names in Amazon S3.
One way to help ensure uniqueness is to prefix your bucket names with the name of your
organization.

The bucket name is visible in the URL that points to the objects that you're going to put in
your bucket. For that reason, choose a bucket name that reflects the objects in the bucket.

Bucket names must comply with the following requirements.

• Can contain lowercase letters, numbers, periods (.), underscores (_), and hyphens (-).

• Must start with a number or letter.

• Must be between 3 and 255 characters long.

• Must not be formatted as an IP address (e.g., 192.168.5.4) .

To conform with DNS requirements, we recommend the following, additional guidelines when
creating bucket names. Bucket names:

• Should not contain underscores (_)

© 2014 ZS Associates
Page - 7
 • Should be between 3 and 63 characters long

• Should not end with a hyphen

• Cannot contain two, adjacent periods

• Cannot contain dashes next to periods (e.g., my-.bucket.com and my.-bucket are
invalid)

Note
Once you create a bucket, you cannot change the name of it. Make sure the bucket name you
choose is appropriate.

4. In the Region box, click the region where you want the bucket to reside.

You should choose a region close to you to optimize latency, minimize costs, or to address
regulatory requirements. Objects stored in a region never leave that region unless you
explicitly transfer them to another region. For more information about regions, see Regions.

In the next step, you have the opportunity to set up logging. Server access logging provides
detailed records for the requests made against your bucket. An access log record contains
details about the request, such as the request type, the resources specified in the request
worked, and the time and date the request was processed. Server access logs are useful for
many applications because they give bucket owners insight into the nature of requests made
by clients not under their control. Amazon S3 delivers access logs to your bucket. By default,
Amazon S3 does not collect server access logs.

5. Do one of the following.

To... Do this...

Create a bucket without setting up logging Click Create

Set up server access logging for the bucket Click Set Up

you're creating Logging

6. Note
7. There is no extra charge for enabling server access logging on an Amazon S3 bucket.
However, any log files the system delivers to you will accrue the usual charges for storage.
(You can delete log files at any time.) We do not assess data transfer charges for delivering
log files to your bucket, but we do charge the normal data transfer rate for accessing the log
files. For more information, go to Amazon S3 Pricing.
8. If you clicked Set Up Logging in the Create a Bucket - Set Up Logging dialog box, do the
following:

© 2014 ZS Associates
Page - 8
 a. Select the Enabled check box.

b. In the Target Bucket box, select the bucket where you want the log files stored.

c. (Optional) In Target Prefix box, specify a prefix for the name of the log files.

Amazon S3 adds the prefix to the log file names when storing them in your bucket.
For example, if you specify the prefix "logs/," all logs stored in the target bucket are
prefixed with logs/, so, all the logs will be stored in the logs folder.

9. Click Create.

If Amazon S3 successfully creates your bucket, the console displays your empty bucket.

© 2014 ZS Associates
Page - 9
 4. Create a Folder

1. Click the bucket in the All Buckets list in which you want to create a folder.

2. Click Create Folder.

3. Under Name, in the box that appears, type a name for the folder, and then click the check mark.

5. Accessing Files through s3CMD

Register for Amazon AWS / S3

Go to Amazon S3 homepage, click on the "Sign up for web service" button in the right column and

work through the registration. You will have to supply your Credit Card details in order to allow

Amazon charge you for S3 usage. At the end you should posses your Access and Secret Keys.

Run s3cmd –configure

You will be asked for the two keys - copy and paste them from your confirmation email or from

your Amazon account page. Be careful when copying them! They are case sensitive and must be

entered accurately or you'll keep getting errors about invalid signatures or similar.

You can optionally enter a GPG encryption key that will be used for encrypting your files before

sending them to Amazon. Using GPG encryption will protect your data against reading by Amazon

staff or anyone who may get access to your them while they're stored at Amazon S3.

Another option to decide about is whether to use HTTPS or HTTP transport for communication with

Amazon. HTTPS is an encrypted version of HTTP, protecting your data against eavesdroppers while

they're in transit to and from Amazon S3.

© 2014 ZS Associates
Page - 10
 Please note: - both the above mentioned forms of encryption are independent on each other and

serve a different purpose. While GPG encryption is protects your data against reading while they

are stored in Amazon S3, HTTPS protects them only while they're being uploaded to Amazon S3

(or downloaded from). There are pros and cons for each and you are free to select either, or, both

or none.

Run s3cmd ls to list all your buckets.

As you have just started using S3 there are no buckets owned by you as of now. So the output will

be empty.

Make a bucket with s3cmd mb s3://my-new-bucket-name

As mentioned above bucket names must be unique amongst _all_ users of S3. That means the

simple names like "test" or "asdf" are already taken and you must make up something more

original. I often prefix my bucket names with my e-mail domain name (logix.cz) leading to a

bucket name, for instance, 'logix.cz-test':

~$ s3cmd mb s3://logix.cz-test
Bucket 'logix.cz-test' created

List your buckets again with s3cmd ls

Now you should see your freshly created bucket

~$ s3cmd ls
2007-01-19 01:41 s3://logix.cz-test

List the contents of the bucket

~$ s3cmd ls s3://logix.cz-test
Bucket 'logix.cz-test':
~$
Upload a file into the bucket
~$ s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml
File 'addressbook.xml' stored as s3://logix.cz-test/addrbook.xml
(123456 bytes)

Note about ACL (Access control lists) — a file uploaded to Amazon S3 bucket can either

be private, that is readable only by you, possessor of the access and secret keys, or public,

readable by anyone. Each file uploaded as public is not only accessible using s3cmd but also has

a HTTP address, URL, that can be used just like any other URL and accessed for instance by web

browsers.
~$ s3cmd put --acl-public --guess-mime-type storage.jpg s3://logix.cz-
test/storage.jpg
File 'storage.jpg' stored as s3://logix.cz-test/storage.jpg (33045
bytes)

© 2014 ZS Associates
Page - 11
 Public URL of the object is: http://logix.cz-
test.s3.amazonaws.com/storage.jpg

Now anyone can display the storage.jpg file in their browser. Cool, eh?

Now we can list the bucket contents again

~$ s3cmd ls s3://logix.cz-test
Bucket 'logix.cz-test':
2008-01-19 01:46 120k s3://logix.cz-test/addrbook.xml
2008-11-14 01:46 32k s3://logix.cz-test/storage.jpg

Retrieve the file back and verify that its hasn't been corrupted
~$ s3cmd get s3://logix.cz-test/addrbook.xml addressbook-2.xml
Object s3://logix.cz-test/addrbook.xml saved as 'addressbook-2.xml'
(123456 bytes)

~$ md5sum addressbook.xml addressbook-2.xml

39bcb6992e461b269b95b3bda303addf addressbook.xml
39bcb6992e461b269b95b3bda303addf addressbook-2.xml

Checksums of the original file matches the one of the retrieved one.

Clean up: delete the object and remove the bucket

~$ s3cmd rb s3://logix.cz-test
ERROR: S3 error: 409 (Conflict): BucketNotEmpty

© 2014 ZS Associates
Page - 12
2 Introduction to the S3 Console
This section provides an overview of the functionality of the AWS Management Console, which is located
at https://console.aws.amazon.com/s3/home.

2.1 S3 Browser for easy file navigation

Buckets View

The Buckets list shows the buckets that you own. The details pane shows bucket properties and the
configuration options.

© 2014 ZS Associates
Page - 13
Objects and Folders View

The Objects and Folders list shows the contents of the Amazon S3 bucket that is indicated. You use folders to
create logical groupings of objects. The details pane shows the properties of the object that is selected in the
Objects and Folders list.

You can use the console to manage all your Amazon S3 resources. You can also use the console to manage
multiple objects at the same time.

© 2014 ZS Associates
Page - 14