The other day I needed to download the contents of a large S3 folder. That is a tedious task in the browser: log into the AWS console, find the right bucket, find the right folder, open the first file, click download, maybe click download a few more times until something happens, go back, open the next file, over and over. Happily, Amazon provides AWS CLI, a command line tool for interacting with AWS. With AWS CLI, that entire process took less than three seconds:
$ aws s3 sync s3:///
Getting set up with AWS CLI is simple, but the documentation is a little scattered. Here are the steps, all in one spot:
You can install AWS CLI for any major operating system:
$ brew install awscli
$ pip install awscli
Download the installer from http://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html
Documentation for the following steps is here.
3. Click on your user name (the name not the checkbox).
4. Go to the Security credentials tab
5.Click Create access key. Don't close that window yet!
6. You'll see your Access key ID. Click "Show" to see your Secret access key.
7. Download the key pair for safe keeping, add the keys to your password app of choice, or do whatever you do to keep secrets safe. Remember this is the last time Amazon will show this secret access key.
Run aws configure and answer the prompts.
Each prompt lists the current value in brackets. On the first run of aws configure you will just see [None]. In the future you can change any of these values by running aws cli again. The prompts will look like AWS Access Key ID [****************ABCD], and you will be able to keep the configured value by hitting return.
$ aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: Default output format [None]:
|ap-northeast-1||Asia Pacific (Tokyo)|
|ap-northeast-2||Asia Pacific (Seoul)|
|ap-south-1||Asia Pacific (Mumbai)|
|ap-southeast-1||Asia Pacific (Singapore)|
|ap-southeast-2||Asia Pacific (Sydney)|
|eu-central-1||EU Central (Frankfurt)|
|eu-west-1||EU West (Ireland)|
|eu-west-2||EU West (London)|
|sa-east-1||South America (Sao Paulo)|
|us-east-1||US East (Virginia)|
|us-east-2||US East (Ohio)|
|us-west-1||US West (N. California)|
|us-west-2||US West (Oregon)|
In the example above, the s3 command's sync command "recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files" (from s3 sync's documentation). I'm able to download an entire collection of images with a simple
aws s3 sync s3://s3.aws-cli.demo/photos/office ~/Pictures/work
But AWS CLI can do much more. Check out the comprehensive documentation at AWS CLI Command Reference.