When to use an AWS S3 VPC endpoint

When to use an AWS S3 VPC Endpoint

What is an S3 VPC endpoint?

To understand what an S3 VPC endpoint is, we first need to know what problem it solves.

Imagine we want to get access to S3 from an AWS resource. In the example below, we have an EC2 instance that needs to copy a file from an S3 bucket:

This works, because:

  • the EC2 instance is in a public subnet, so has access to the internet
  • therefore the EC2 instance can reach the AWS S3 URL to copy the file from the S3 bucket

Public subnets
A public subnet is simply one that has a route to the internet. In the case of AWS, this means it has a route table with a route to an internet gateway

The problem with S3 access from a private subnet

Where this starts to fall down, though, is when we need to access S3 from an EC2 instance in a private subnet, as in the example below:

This doesn’t work, because:

  • the EC2 instance is in a private subnet, so has no internet access
  • therefore the EC2 instance can’t reach the AWS S3 URL, and the request will time out

S3 VPC endpoints solve this problem

An S3 VPC endpoint provides a way for an S3 request to be routed through to the Amazon S3 service, without having to connect a subnet to an internet gateway.

The S3 VPC endpoint is what’s known as a gateway endpoint. It works by adding an entry to the route table of a subnet, forwarding S3 traffic to the S3 VPC endpoint. The other type of gateway endpoint is for DynamoDB.

The image below shows a route table which has the S3 endpoint included. We have a route for requests with a destination s3.eu-west-1.amazonaws.com to target the VPC endpoint. Therefore any S3 requests will be routed through to S3.

Demo: setting up a public & private subnets and EC2 instances

In order to demonstrate an S3 VPC endpoint working, and solving the problem highlighted above, we’ll setup the following AWS resources:

  1. a public and private subnet
  2. an EC2 instance in both subnets. We need an EC2 instance in the public subnet so we can access the one in the private subnet i.e. we’re using it as a bastion.
  3. the EC2 instances will have an IAM role associated with them that allows S3 access

Then we’ll SSH into the EC2 instance on the private subnet and see that we can’t make an AWS Command Line Interface (CLI) S3 request.

Finally, we’ll add an S3 VPC endpoint then see that it does provides access to S3, as below:

A public and private subnet

A public subnet is simply a subnet that has a route to the internet, and a private subnet doesn’t.

We need a public subnet into which we’re going to deploy an EC2 instance. Below we can see this subnet has a route to an internet gateway.

And here’s the private subnet, without an route to an internet gateway.

AWS by default creates public subnets for you in your default VPC. If you don’t have a private subnet, you’ll need to add one to follow along with this example. To learn more, you can follow Step 3: Create Additional Subnets in this AWS tutorial.

Create a role for S3 access from EC2 instances

In order to call the S3 service, our EC2 instances will need to have an appropriate role configured. Let’s create that first, so go to Services > IAM > Roles, and select Create Role:

Select what service this role should apply to, by selecting AWS service and the EC2, then click Next:

Now we can select what permissions to provide the role. Search for s3, select AmazonS3FullAccess, then click Next.

Skip over the tags page by also clicking Next. Give the role a name such as s3FullAccess then click Create role.

Launching an EC2 instance into a public subnet

Go to Services > EC2 and click the Launch instance button. Select an Amazon Machine Image (AMI) to use, such as Amazon Linux 2 AMI (HVM), SSD Volume Type, then click Select.

Since we need very little compute power, choose t2.micro, then click Next:

Here we need to select the target VPC and subnet. We’ll be putting this EC2 instance into a public subnet, so select that from the drop down list. For the IAM role field, select the s3FullAccess role we created earlier. Click Next:

On the Add Storage page just click Next. Then on the Add Tags page we’ll add a Name tag with value public instance, and click Next:

On the Configure Security Group page accept the default which allows SSH access, select Review and Launch, then Launch. The popup that appears allows you to choose an existing key pair or create a new one. This key pair will be used for SSH access, so be sure to save the .pem file for later:

Your EC2 instance will be created in the background.

Launching an EC2 instance into a private subnet

Follow the same steps as in the Launching an EC2 instance into a public subnet section, but with two differences:

  1. On the Configure Instance Details page, select the private subnet in your VPC (as well as the s3FullAccess role)
  2. On the Add Tags page add the Name tag with value private instance

After they’ve loaded, on the Services > EC2 > Instances page you should now have the following two instances:

Note down the public IP address of your public instance, and the private IP address of your private instance. We’ll need these later for SSH access.

Demo: adding an S3 VPC endpoint

To first demonstrate the problem that the S3 VPC endpoint is going to solve, let’s SSH into our public instance, then our private instance, then try to do an S3 AWS CLI command.

SSH to the private instance

First, don’t forget to add your key using ssh-add. Then we’ll execute ssh -A on the instance:

ssh-add <pem-file-location>
ssh -A ec2-user@<public-instance-ip>

ssh -A
The -A flag forwards your authentication details, meaning that we’ll be able to use the SSH key again to jump to another host

Now we’ll hop over to our private instance, using its private IP. 🦘

ssh ec2-user@<private-instance-ip>

Failing to run an S3 AWS CLI command

Since Amazon Linux 2 comes with the AWS CLI installed by default, let’s try running aws s3 ls --region <region>:

This hangs and eventually fails with a timeout. To summarise what’s happening here, remember that:

  1. Our instance does have an IAM role attached with full S3 access, so this isn’t a permissions problem
  2. Our instance is in a private subnet with no route to the internet, so the AWS CLI can’t establish a connection to the S3 service

Time to add an S3 VPC endpoint? I think so! ✅

Region flag
The --region flag is needed in the aws s3 ls call above since the VPC endpoint will be for a specific region. Specify the same region as your VPC.

Adding an S3 VPC endpoint

Navigate to Services > VPC > Endpoints, and select the tempting big blue Create Endpoint button:

Now we have to tell AWS what type of endpoint we want to create. Search for s3, and select the com.amazonaws.<region>.s3 Service Name:

Next up we need to select where we want the VPC endpoint to apply. Select your VPC, then select your private route table (the one that is associated with the private subnet). Leave the policy as Full Access, meaning that the S3 endpoint will allow requests through to the S3 service for any AWS account.

Select Create endpoint:

You will see this success page:

In the background AWS will have updated your route table to include the route to the S3 endpoint. Go to Services > VPC > Route Tables and select the route table to which you just added the S3 endpoint.

If you click on Routes you can see the new route that was added (this may take a minute to appear):

Trying the S3 AWS CLI command again

Back on our private instance, let’s try that aws s3 ls --region <region> command again:

Success! The AWS CLI S3 command has successfully sent a request via the VPC endpoint to the S3 service to list out our buckets.

Final thoughts

We’ve seen that accessing S3 from a private subnet that doesn’t have any internet access is possible thanks to the S3 VPC endpoint. Once created, it automatically updates the route table of the private subnet to allow S3 requests to reach the S3 service.

You may be thinking as an alternative to this you could allow access to the internet from your private EC2 instance, by setting up an AWS NAT Gateway in your public subnet. This is true, but bear in mind the additional cost of the NAT Gateway. An AWS S3 VPC endpoint, on the other hand, is free.

From a security standpoint, the S3 VPC endpoint is a robust solution because you’re only allowing traffic out to the S3 service specifically, and not the whole internet. If this fits in with your use case, then the S3 VPC endpoint could be the way to go.

Resources

AWS Gateway Endpoints
Check out the AWS docs on this subject

AWS VPCs
Check out my other article on When to create different subnets in AWS VPCs

VIDEO
If you prefer to learn in video format then check out the accompanying video below. It’s part of the Tom Gregory Tech YouTube channel.

When to use an AWS S3 VPC endpoint

17 thoughts on “When to use an AWS S3 VPC endpoint

  1. Hey Tom,
    Great article with good explanation for the VPC end points with diagrams . I would request you to write more articles for the AWS, since your thoughts are explained in a clear, easy to understand . Thanks alot.

    1. Hi Chandra. Thanks so much for the feedback. More articles on the way soon 🙂 What kind of topics are you interested in?

      1. This is really great! The “you can still do this with a NAT gateway” point CLEARED up the confusion I had about the network logic difference between the two approaches.

  2. Hi Tom
    nice work!
    One question. I understand that this VPC endpoint is useful when our instances in private subnets dont’ have access and then we are saving money, we save a NAT Gateway so its cool.

    But if we do the same in a public subnet, does it make sense? I mean, if we do the VPC endpoint in the route table of the public subnet, this traffic won’t go over our internet gateway and will remain in the AWS network?

    Or Im nuts?

    Thanks

    1. Hi Ruben. Good question!

      I suppose if there was a security requirement that all S3 traffic remain internal to your VPC then the S3 VPC endpoint would make sense, even in the public subnet.

      If you don’t have such a requirement, then from the public subnet you might want just to go out via the Internet Gateway for S3 traffic. It keeps your infrastructure a little bit simpler.

  3. Hi Tom,

    One thing, If I create a EC2 in the public subnet and then create a VPC endpoint associating it with the public subnet. Will I be charged for the GET,POST requests I make to the S3 bucket from that EC2.

    Ps:- I already got charged 100$ which I think I could save via VPC endpoints for S3 😉

    1. Hi Bhanu,

      As far as I understand there are no transfer costs between EC2 and S3 within the same region. However there are charges for storage and number of requests.

      What was the $100 charge for?

  4. Hi Tom, Thanks for the great article. You are doing a great work for people like me, who are still learning 😉
    Just a question, what will happen if other services are also using the same s3 bucket?
    Will they all be routed via vpc endpoint? (i guess they would)
    Also if my ec2 has also internet access,is it possible to verify if its connecting via vpc endpoint or internet?

    Thanks again _/\_

    1. Hi Prashant. Thanks for saying that 🙂 and good questions.

      1) Other services will also use the S3 VPC endpoint as long as they’re running inside a subnet where the VPC endpoint is configured in the route table

      2) I have these ideas of how you could test that your EC2 instance with internet access is using the S3 VPC endpoint:
      a) temporarily restrict all outbound traffic, except the traffic to the VPC endpoint. You can do this using a security group.
      b) temporarily update your route table to disable internet access. In a private subnet this would involved removing the route to the NAT Gateway.

      There’s probably more ways I didn’t think of, so let me know if you find anything else.

  5. Nice and Informatic article Tom. Thanks
    Our business case is to extract data from a Vendor S3 bucket and load into AWS EMR. We want the data transfer does not go through the public internet at all. I think the solution you have demonstrated using S3 VPC endpoint will help us.

    Also there are S3 buckets which are part of our own account, even to connect to our own buckets we don’t want traffic to go through the public internet.

    Can we also restrict the S3 policy to permit only connections from a specific VPC, so that no other VPC can access S3 buckets.

    Please suggest your thoughts. Thanks alot

    1. Hi Arjun. That’s a very interesting question. You can use conditions in your S3 bucket policy. The ones that might be applicable are:

      • aws:SourceVpc – restrict access from a specific VPC
      • aws:SourceVpce – restrict access from a specific VPC endpoint

      It all depends on how restrictive you want to be. AWS have some examples here.

  6. I tried following the steps mentioned above:
    But the private ec2 instance is unable to query S3

    I configured credentials in ~/.aws/credentials and only region in ~/.aws/config files

    If I provide role_arn, still it fails with error:
    2021-02-18 09:30:00,858 – MainThread – urllib3.connectionpool – DEBUG – Starting new HTTPS connection (4): sts.amazonaws.com:443

  7. Hi,
    I have a question, can we say that using the VPC endpoint from the performance point of view is faster in transfer data between EC2 and S3 than going the traffic through an Internet gateway (IGW)?
    regards.

    1. Hi AhmadR. This is an interesting question. I don’t have any data on this, but when using the VPC endpoint your traffic stays in the AWS network and doesn’t cross the public internet. This in theory means there should be a performance benefit.

  8. Hi Tom, great article. easy to understand.
    my use case is to access s3 from another aws account. in my account, there is already s3 vpc endpoint configured and endpoint policy is not restrcitive.
    further, role is already configured to have access to s3 bucket in other account.
    question is, can same S3 vpc endpoint can be used to access bucket from other account ?
    or do i need to do make any further changes ?

    1. Hi Amardeep. Thanks for the question. The VPC endpoint can by default access any bucket in the same region. If you’re also asking about sharing an S3 VPC endpoint between accounts, I don’t know of a way to do that. Since the VPC endpoint is free, you can just create another one in the other account.

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top

Get the newsletter

Found this article helpful? Subscribe for monthly updates.

✅ All of my latest articles for the month
✅ Access to video tutorials
✅ Exclusive tips not found on my website