AWS CLI

The AWS CLI is Amazon’s command-line tool for S3. With a separate zata profile, you can list, upload, download and share files in your Zata buckets, and keep your AWS settings untouched.

Before you start

• A bucket. See Create a bucket.
• An access key and secret key from the same region as the bucket. See Access keys.
• AWS CLI version 2.13 or newer. Check with aws --version.

Connection settings

Install the AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip -q awscliv2.zip
sudo ./aws/install
aws --version

Set up a Zata profile

aws configure --profile zata

aws configure set endpoint_url https://idr01.zata.ai --profile zata
aws configure set s3.addressing_style path --profile zata
aws configure set request_checksum_calculation when_required --profile zata
aws configure set response_checksum_validation when_required --profile zata

These commands work the same in a Linux, macOS or Windows terminal. They save the settings in ~/.aws/config and ~/.aws/credentials (on Windows, in %USERPROFILE%\.aws\).

Test it

Replace my-bucket with your bucket name:

aws s3 ls s3://my-bucket --profile zata
aws s3 cp myfile.txt s3://my-bucket/ --profile zata

aws s3 ls --profile zata lists your buckets. A sub-user’s key shows no buckets there; use the bucket name directly.

Everyday commands

# Download a file
aws s3 cp s3://my-bucket/myfile.txt ./myfile.txt --profile zata
# Upload a folder
aws s3 cp ./my-folder s3://my-bucket/my-folder/ --recursive --profile zata
# Upload only new and changed files
aws s3 sync ./my-folder s3://my-bucket/my-folder/ --profile zata
# Delete a file
aws s3 rm s3://my-bucket/myfile.txt --profile zata
# Make a download link that works for 1 hour (3600 seconds)
aws s3 presign s3://my-bucket/myfile.txt --expires-in 3600 --profile zata
# Check that a bucket exists in this region
aws s3api head-bucket --bucket my-bucket --profile zata

aws s3 cp uploads files larger than 8 MB in parts (multipart) by itself. aws s3 presign makes download links only; for upload links, use an SDK such as Python (boto3).

Test a multipart upload step by step

aws s3 cp does all of this by itself. Doing it by hand shows which step fails when another app cannot upload large files. Every part except the last must be at least 5 MB.

head -c 12M /dev/urandom > bigfile.bin
split -b 5M -d bigfile.bin part-

UPLOAD_ID=$(aws s3api create-multipart-upload --bucket my-bucket --key test/bigfile.bin --query UploadId --output text --profile zata)
echo "$UPLOAD_ID"

aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 1 --body part-00 --upload-id "$UPLOAD_ID" --profile zata
aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 2 --body part-01 --upload-id "$UPLOAD_ID" --profile zata
aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 3 --body part-02 --upload-id "$UPLOAD_ID" --profile zata

aws s3api list-parts --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --query '{Parts: Parts[].{PartNumber: PartNumber, ETag: ETag}}' --output json --profile zata > parts.json
cat parts.json

aws s3api complete-multipart-upload --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --multipart-upload file://parts.json --profile zata

aws s3api abort-multipart-upload --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --profile zata

If the complete step fails, Zata removes that upload: start again with a new upload ID.

Unfinished uploads keep using storage until they are completed or cancelled. To find them:

aws s3api list-multipart-uploads --bucket my-bucket --profile zata

A sub-user’s key may get AccessDenied on this list; the bucket owner’s key works.

If these steps work with the AWS CLI but another app still fails on large files, that app most likely sends the default checksums. See Tools built on the AWS SDK.

Troubleshooting

aws configure list --profile zata shows which keys and settings the profile really uses. For other errors, see Signature errors, Access denied and Wrong endpoint or region.