Google Cloud Platform - Ways of Uploading Data to GCS

Before you can harness the power of the cloud, to serve your content, you have to get your data into it. In this article, we will look into all the different ways you can upload data. We all know that Google Cloud Storage (GCS) can serve your binary assets to users worldwide at high speed and low price. But before you can do any of that, you need to get your data into the Google Cloud, which can be done in a few specific ways. Let's look into them in detail.

1. Using Google Cloud Console:

This is the easiest option. This provides you with an in-browser experience where you can easily click to create buckets and folders and then choose or drag and drop the files from your local machine to upload.

2. Using GSUTIL:

This is the Command line tool approach. While the console is fine for easy uploads in simplistic serving situations but for production environments, you often need an automated command-line solution. For this, the GSUTIL tool can be used which is a Python application that lets you access cloud storage from the command line. It provides you with the ability to all sorts of operations like creating buckets, moving objects, or even editing metadata. To use it simply run the GSUTIL program with a myriad of command-line options.

For example, the below command uploads a directory of files from your local machine to the GCS bucket using parallel upload.

gsutil -m cp -r dir gs://my-bucket

The below command lists out specific objects that have a version-specific URL using a wild card:

gsutil ls -a gs://bucket/images/*.jpg

3. Using Client Libraries:

At some point, you might need an interface with the GCS directly from your code rather than going out to a command-line option. You can include the client libraries in your code and call a simple API to get data into a bucket or a folder and regardless of your language it probably has you covered.

4. Using JSON and XML:

If none of the above suits your need there are always the good old JSON and XML APIs which you can let kick off an HTTP POST to upload data directly to the bucket or a folder. It's a bit more complex but it's there in case it fits your use case.

5. Using GCS Transfer Appliance

Now, for the folks with lots and lots of data, it's worth noting that it might not be feasible to upload all of that data from your on-premise systems directly to the Google cloud. For this, the GCS Transfer Appliance can be used. Google ships you a fancy device directly to your location and you can get connected and transfer your files at super high speed.