s5cmd

s5cmd is a very fast command-line tool for S3. It copies thousands of files in parallel, which makes it a good choice for large folders and migrations. It has no setup wizard: you give it the keys in environment variables and the Zata endpoint on each command.

Before you start

• A bucket. See Create a bucket.
• An access key and secret key from the same region as the bucket. See Access keys.

Connection settings

The examples below use Indore. For a bucket in Mumbai, use https://bom01.zata.ai.

Install s5cmd

Download the file for your system from the s5cmd releases page. On Linux (64-bit):

curl -sL -o s5cmd.tar.gz https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-64bit.tar.gz
tar xzf s5cmd.tar.gz s5cmd
sudo mv s5cmd /usr/local/bin/
s5cmd version

Give s5cmd your keys

In the terminal you will use:

export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY

To avoid typing the endpoint on every command, also set:

export S3_ENDPOINT_URL=https://idr01.zata.ai

To keep the keys in a file instead, put them in ~/.aws/credentials and add --profile zata to each command:

[zata]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Test it

Replace my-bucket with your bucket name:

s5cmd --endpoint-url https://idr01.zata.ai ls s3://my-bucket/

Copy many files

Put the source in single quotes, so that s5cmd (not the shell) expands the *. Sub-folders are kept.

Upload a folder:

s5cmd --endpoint-url https://idr01.zata.ai cp '/path/to/folder/*' s3://my-bucket/folder/

Upload only new or changed files (nothing is deleted in the bucket):

s5cmd --endpoint-url https://idr01.zata.ai sync '/path/to/folder/*' s3://my-bucket/folder/

Download a folder:

s5cmd --endpoint-url https://idr01.zata.ai cp 's3://my-bucket/folder/*' ./folder/

Large files are uploaded in parts automatically.

Speed settings

Troubleshooting

For other errors, see Access denied and Wrong endpoint or region.