From AWS S3

Use rclone to copy objects from an AWS S3 bucket to a Zata bucket. rclone reads from AWS and writes to Zata in one command. The data passes through the computer that runs rclone.

Before you start

• A Zata bucket, and a Zata access key from the same region. See Create a bucket and Access keys.
• An AWS access key that can read the source bucket. In the AWS console, open IAM → Users → your user → Security credentials → Create access key. A user with the AWS managed policy AmazonS3ReadOnlyAccess is enough.
• The AWS region of the source bucket, for example ap-south-1 (Mumbai).
• rclone on a computer with a fast internet connection. Install the latest version from rclone.org; on Linux run curl https://rclone.org/install.sh | sudo bash. For a large bucket, a cloud server such as a NeevCloud virtual machine is faster than an office connection.

1. Add the AWS remote

Replace ap-south-1 with the region of your AWS bucket.

rclone config create aws s3 provider=AWS access_key_id=YOUR_AWS_ACCESS_KEY secret_access_key=YOUR_AWS_SECRET_KEY region=ap-south-1

2. Add the Zata remote

For a bucket in Mumbai, use https://bom01.zata.ai as the endpoint.

rclone config create zata s3 provider=Other access_key_id=YOUR_ACCESS_KEY secret_access_key=YOUR_SECRET_KEY region=us-east-1 endpoint=https://idr01.zata.ai force_path_style=true no_check_bucket=true sign_accept_encoding=false

Two of these settings are needed on Zata:

To set up the remote step by step instead, see rclone on Linux.

3. Check both remotes

rclone lsd aws:
rclone ls zata:my-bucket

The first command lists your AWS buckets. The second lists the Zata bucket, and shows nothing while it’s empty. With a sub-user key, rclone lsd zata: shows no buckets, so always use the bucket name.

4. Copy the data

Replace my-aws-bucket with the AWS bucket and my-bucket with the Zata bucket.

rclone copy aws:my-aws-bucket zata:my-bucket --progress

• copy never deletes anything. It skips files that are already the same in Zata, so if it stops halfway, run the same command again.
• For a bucket with millions of objects, add --fast-list --transfers 16 --checkers 32.
• If your apps still write to AWS during the migration, run the copy again when you switch over to Zata, to pick up the latest files.

5. Check the copy

rclone check aws:my-aws-bucket zata:my-bucket --one-way

It ends with 0 differences found when every object is in Zata with the same content. Objects uploaded in parts have no MD5 to compare. To compare their content too, add --download (slower, and it downloads the files from AWS again).

What rclone doesn’t copy

• Old versions. rclone copies the current version of each object only.
• Bucket settings. Versioning, lifecycle rules, bucket policies and CORS rules stay in AWS. Set them up again in Zata. If you need Object Lock, turn it on when you create the Zata bucket, before you copy: you can’t add it later.
• Archived objects. Objects in S3 Glacier storage classes must be restored in AWS before rclone can read them. Zata has one storage class, so they arrive as normal objects.

Troubleshooting