Disclaimer

AWS S3 - Private and Public Access

aws security

Intro

As I noticed yet another S3 bucket incident where data was not only accessed, but JavaScript code was added, I’ve decided to write a post describing the steps required to create a bucket with private access, and (intentionally) make the objects public using a Content Delivery Network(CDN).

I also added a few words regarding VPC Endpoints, and detection of non-compliant resources (i.e. buckets with public access).

Working with default settings

Public access is disabled by default

In recent years, AWS has went out of its way to make it difficult for users to accidentally configure public access for an S3 bucket.

So I created an S3 bucket, accepting the default settings.

If I would store sensitive data in this bucket, I would consider using encryption, however this is not the case, nor is it the object of this experiment.

Testing public access

I’ll upload a photo of a bucket spilling its contents and try to access it from the browser.

Public access is denied - as expected, so unless you want to make the objects public there’s not much else to do.

Using a VPC Endpoint

A VPC endpoint offers private connectivity to services hosted in AWS, from within your VPC without using an internet gateway.

One of the benefits is that access can be restricted to a specific VPC Endpoint rather than using IP address ranges.

This could be used to allow write access from within the VPC.

Another benefit is that it can help you reduce costs, e.g. data transfer charges for NAT gateways

More info in the documentation regarding Gateway VPC endpoints and Endpoints for Amazon S3.

What if I want to make the content public?

I will always use a Content Delivery Network (CDN) instead of exposing the S3 bucket directly because of the following.

Lower cost (at least for some CDNs)

You can find the cost of serving objects by looking at Data Transfer OUT From Amazon S3 To Internet S3 pricing - in the Data Transfer tab.

As there are many CDNs, I cannot claim that all of them are cheaper, so I picked 2: AWS Cloudfront and Cloudflare.

AWS Cloudfront

Data transfer between AWS S3 and AWS Cloudfront is free, so I would only pay for Cloudfront usage. More details on the Cloudfront pricing page

Cloudflare

Data transfer between AWS S3 and other CDNs like Cloudflare is not free, however I will only pay for objects which are not already cached by the CDN.

As Cloudflare has a free plan for personal use, the only cost I wil incur is the transfer from S3 to Cloudflare for objects which are not cached. More about how Cloudflare works

Increased performance

Even if you follow the S3 performance best practices you could run into some rate limits.

The objects contained in S3 buckets are stored in an AWS region, therefore the users in distant parts of the globe will experience increased latency. More info in the Performance Guidelines for Amazon S3

Allowing public access through Cloudflare

I will create a bucket named s3-demo.testbox.blue which I plan to access it using the domain with the same name. More details in the AWS documentation

Bucket Policy

As I want to only allow Cloudflare to access my S3 bucket directly, I will do this via a Bucket Policy.

In the code below I left only one IP in the list to keep it shorter, the full list and a more detailed explanation can be found in this Cloudflare article

The action is s3:GetObject, so the CDN will only be able to retrieve files from the S3 bucket.

I also took an optional step of preventing the removal of this bucket until the statement denying s3:DeleteBucket is removed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:DeleteBucket",
            "Resource": "arn:aws:s3:::s3-demo.testbox.blue"
        },
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::s3-demo.testbox.blue/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "198.41.128.0/17"
                    ]
                }
            }
        }
    ]
}

If I try to save this policy I will not be allowed with the current configuration.

First, public access must be allowed.

I also had to confirm this action

Now I could save the policy.

Static website hosting

Next I enabled static hosting for the bucket.

CNAME for S3 bucket

I will add a CNAME record s3-demo for the testbox.blue domain pointing to s3-demo.testbox.blue.s3-website-eu-west-1.amazonaws.com

Testing public access (again)

The uploaded photo is now accessible using the custom domain.

The photo is not accessible from S3, not even when using the website URL, which was the expected outcome.

Some more comments

Avoiding “Objects can be public” state

Creating a new bucket, and unchecking Block all public access will display the bucket’s access as Objects can be public

Now I could upload another photo, and update the object’s Access Control List (ACL) in order to make it public

The object is now publicly accessible

However if someone would look at the list of S3 buckets, it wouldn’t be clear if there are any public objects or not.

The security tools I’ve seen so far do not report this issue, so it’s best to avoid this gray area.

Setting up notifications/remediation actions

AWS Config has the s3-bucket-public-read-prohibited, and support for remediation actions.

If I wouldn’t want to use AWS Config, then one idea would be to use GetBucketPolicyStatus, for example from a Lambda function which runs regularly, then send a notification, or even implement the remediation action as another Lambda.

Conclusion

In my opinion, the best way to prepare for such an incident is to assume that it will happen sooner or later and you need to be prepared - get notified, and if possible remediate the problem automatically.