My Digital Garden

Transferring files between S3 buckets in different accounts

Transferring files between S3 buckets in different accounts

specifics are for Ubuntu on WSL2

  1. Make note of arn reference for source and destination buckets

  2. Make note of arn reference for user that owns the destinmation bucket

  3. log into WSL2

  4. install or update AWS CLI

  5. Create an IAM user and role in the destination AWS account

    • create user, note arn, access key and secret key

  6. create IAM-based Identity Policy, named S3MigrationPolicy:

    {
 "Version": "2012-10-17",
 "Statement": [
     {
         "Effect": "Allow",
         "Action": [
             "s3:ListBucket",
             "s3:GetObject",
             "s3:GetObjectTagging",
             "s3:GetObjectVersion",
             "s3:GetObjectVersionTagging"
         ],
         "Resource": [
             "arn:aws:s3:::amazon-s3-demo-source-bucket",
             "arn:aws:s3:::amazon-s3-demo-source-bucket/*"
         ]
     },
     {
         "Effect": "Allow",
         "Action": [
             "s3:ListBucket",
             "s3:PutObject",
             "s3:PutObjectAcl",
             "s3:PutObjectTagging",
             "s3:GetObjectTagging",
             "s3:GetObjectVersion",
             "s3:GetObjectVersionTagging"
         ],
         "Resource": [
             "arn:aws:s3:::amazon-s3-demo-destination-bucket",
             "arn:aws:s3:::amazon-s3-demo-destination-bucket/*"
         ]
     }
   ]
 }

  7. create an IAM role named S3MigrationRole by using the following trust policy, and then attach the previously created S3MigrationPolicy in the portal.

    (see Create a role to give permissions to an IAM user)

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<destination_account>:user/<user_name>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

    remember to connect this to the policy in the portal while you are creating the role

  8. Create and attach the S3 bucket policy in the source account (or in reality, this will probably be an update). This example grants access to the role previously set up in the destination account, but in some simplified circumstances it may be sufficient to just grant access to a user id.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::<destination_account>:role/<RoleName>"},
            "Action": ["s3:ListBucket",
                      "s3:GetObject",
                      "s3:GetObjectTagging",
                      "s3:GetObjectVersion",
                      "s3:GetObjectVersionTagging"
                      ],
            "Resource": [
                "arn:aws:s3:::amazon-s3-demo-source-bucket/*",
                "arn:aws:s3:::amazon-s3-demo-source-bucket"
            ]
        }
    ]
}

  9. Copy the data

    • if you are using role-based access, use the CLI to assume the role:

      aws sts assume-role \
  --role-arn "arn:aws:iam::<destination_account>:role/S3MigrationRole" \
  --role-session-name AWSCLI-Session

    • copy the files

      aws s3 cp s3://amazon-s3-demo-source-bucket/ \
  s3://amazon-s3-demo-destination-bucket/ \
  --recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

    • for usbsequent updates use sync command

      aws s3 sync s3://amazon-s3-demo-source-bucket/ \
  s3://amazon-s3-demo-destination-bucket/ \
  --recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

See Also