What is an S3 VPC endpoint?
To understand what an S3 VPC endpoint is, we first need to know what problem it solves.
Imagine we want to get access to S3 from an AWS resource. In the example below, we have an EC2 instance that needs to copy a file from an S3 bucket:
This works, because:
- the EC2 instance is in a public subnet, so has access to the internet
- therefore the EC2 instance can reach the AWS S3 URL to copy the file from the S3 bucket
Public subnets
A public subnet is simply one that has a route to the internet. In the case of AWS, this means it has a route table with a route to an internet gateway
The problem with S3 access from a private subnet
Where this starts to fall down, though, is when we need to access S3 from an EC2 instance in a private subnet, as in the example below:
This doesn’t work, because:
- the EC2 instance is in a private subnet, so has no internet access
- therefore the EC2 instance can’t reach the AWS S3 URL, and the request will time out
S3 VPC endpoints solve this problem
An S3 VPC endpoint provides a way for an S3 request to be routed through to the Amazon S3 service, without having to connect a subnet to an internet gateway.
The S3 VPC endpoint is what’s known as a gateway endpoint. It works by adding an entry to the route table of a subnet, forwarding S3 traffic to the S3 VPC endpoint. The other type of gateway endpoint is for DynamoDB.
The image below shows a route table which has the S3 endpoint included. We have a route for requests with a destination s3.eu-west-1.amazonaws.com to target the VPC endpoint. Therefore any S3 requests will be routed through to S3.
Demo: setting up a public & private subnets and EC2 instances
In order to demonstrate an S3 VPC endpoint working, and solving the problem highlighted above, we’ll setup the following AWS resources:
- a public and private subnet
- an EC2 instance in both subnets. We need an EC2 instance in the public subnet so we can access the one in the private subnet i.e. we’re using it as a bastion.
- the EC2 instances will have an IAM role associated with them that allows S3 access
Then we’ll SSH into the EC2 instance on the private subnet and see that we can’t make an AWS Command Line Interface (CLI) S3 request.
Finally, we’ll add an S3 VPC endpoint then see that it does provides access to S3, as below:
A public and private subnet
A public subnet is simply a subnet that has a route to the internet, and a private subnet doesn’t.
We need a public subnet into which we’re going to deploy an EC2 instance. Below we can see this subnet has a route to an internet gateway.
And here’s the private subnet, without an route to an internet gateway.
AWS by default creates public subnets for you in your default VPC. If you don’t have a private subnet, you’ll need to add one to follow along with this example. To learn more, you can follow Step 3: Create Additional Subnets in this AWS tutorial.
Create a role for S3 access from EC2 instances
In order to call the S3 service, our EC2 instances will need to have an appropriate role configured. Let’s create that first, so go to Services > IAM > Roles, and select Create Role:
Select what service this role should apply to, by selecting AWS service and the EC2, then click Next:
Now we can select what permissions to provide the role. Search for s3, select AmazonS3FullAccess, then click Next.
Skip over the tags page by also clicking Next. Give the role a name such as s3FullAccess then click Create role.
Launching an EC2 instance into a public subnet
Go to Services > EC2 and click the Launch instance button. Select an Amazon Machine Image (AMI) to use, such as Amazon Linux 2 AMI (HVM), SSD Volume Type, then click Select.
Since we need very little compute power, choose t2.micro, then click Next:
Here we need to select the target VPC and subnet. We’ll be putting this EC2 instance into a public subnet, so select that from the drop down list. For the IAM role field, select the s3FullAccess role we created earlier. Click Next:
On the Add Storage page just click Next. Then on the Add Tags page we’ll add a Name tag with value public instance, and click Next:
On the Configure Security Group page accept the default which allows SSH access, select Review and Launch, then Launch. The popup that appears allows you to choose an existing key pair or create a new one. This key pair will be used for SSH access, so be sure to save the .pem file for later:
Your EC2 instance will be created in the background.
Launching an EC2 instance into a private subnet
Follow the same steps as in the Launching an EC2 instance into a public subnet section, but with two differences:
- On the Configure Instance Details page, select the private subnet in your VPC (as well as the s3FullAccess role)
- On the Add Tags page add the Name tag with value private instance
After they’ve loaded, on the Services > EC2 > Instances page you should now have the following two instances:
Note down the public IP address of your public instance, and the private IP address of your private instance. We’ll need these later for SSH access.
Demo: adding an S3 VPC endpoint
To first demonstrate the problem that the S3 VPC endpoint is going to solve, let’s SSH into our public instance, then our private instance, then try to do an S3 AWS CLI command.
SSH to the private instance
First, don’t forget to add your key using ssh-add. Then we’ll execute ssh -A on the instance:
ssh-add <pem-file-location> ssh -A ec2-user@<public-instance-ip>
ssh -A
The -A flag forwards your authentication details, meaning that we’ll be able to use the SSH key again to jump to another host
Now we’ll hop over to our private instance, using its private IP. 🦘
ssh ec2-user@<private-instance-ip>
Failing to run an S3 AWS CLI command
Since Amazon Linux 2 comes with the AWS CLI installed by default, let’s try running aws s3 ls --region <region>:
This hangs and eventually fails with a timeout. To summarise what’s happening here, remember that:
- Our instance does have an IAM role attached with full S3 access, so this isn’t a permissions problem
- Our instance is in a private subnet with no route to the internet, so the AWS CLI can’t establish a connection to the S3 service
Time to add an S3 VPC endpoint? I think so! ✅
Region flag
The --region flag is needed in the aws s3 ls call above since the VPC endpoint will be for a specific region. Specify the same region as your VPC.
Adding an S3 VPC endpoint
Navigate to Services > VPC > Endpoints, and select the tempting big blue Create Endpoint button:
Now we have to tell AWS what type of endpoint we want to create. Search for s3, and select the com.amazonaws.<region>.s3 Service Name:
Next up we need to select where we want the VPC endpoint to apply. Select your VPC, then select your private route table (the one that is associated with the private subnet). Leave the policy as Full Access, meaning that the S3 endpoint will allow requests through to the S3 service for any AWS account.
Select Create endpoint:
You will see this success page:
In the background AWS will have updated your route table to include the route to the S3 endpoint. Go to Services > VPC > Route Tables and select the route table to which you just added the S3 endpoint.
If you click on Routes you can see the new route that was added (this may take a minute to appear):
Trying the S3 AWS CLI command again
Back on our private instance, let’s try that aws s3 ls --region <region> command again:
Success! The AWS CLI S3 command has successfully sent a request via the VPC endpoint to the S3 service to list out our buckets.
Final thoughts
We’ve seen that accessing S3 from a private subnet that doesn’t have any internet access is possible thanks to the S3 VPC endpoint. Once created, it automatically updates the route table of the private subnet to allow S3 requests to reach the S3 service.
You may be thinking as an alternative to this you could allow access to the internet from your private EC2 instance, by setting up an AWS NAT Gateway in your public subnet. This is true, but bear in mind the additional cost of the NAT Gateway. An AWS S3 VPC endpoint, on the other hand, is free.
From a security standpoint, the S3 VPC endpoint is a robust solution because you’re only allowing traffic out to the S3 service specifically, and not the whole internet. If this fits in with your use case, then the S3 VPC endpoint could be the way to go.
Resources
AWS Gateway Endpoints
Check out the AWS docs on this subject
AWS VPCs
Check out my other article on When to create different subnets in AWS VPCs
VIDEO
If you prefer to learn in video format then check out the accompanying video below. It’s part of the Tom Gregory Tech YouTube channel.
Hi Tom,
Good explanation, specially diagram, it cleared few doubts about endpoint. I have one specific requirement that one bucket needs to be made publicly available but only for all users on company network. That is if I have connected through company’s vpn, access key or any authorization should not be required for that bucket.
How can I achieve that?
This is an excellent article and makes it easy to understand! Our VPC does not have an IGW, and we were running into a problem with deploying some services. The error was not helpful at all. Amazon had to do a lot of digging on their end because they didn’t know why we were experiencing the error. They discovered that the service needed to reach an Amazon S3 bucket for some patches, and because we don’t have an IGW, it was failing with a generic “service failed” message. They said the fix was an S3 VPC endpoint. You explained it perfectly in the “we need to know what problem it solves.”
Thank you for the article, it makes sense now!
Dear Sir,
Thank you so much for sharing your knowledge in public.
I have a small doubt is what is the bestway to connect & mount S3 bucket to my EC2 centos Asterisk instance? Purpose is to directly store recordings to S3 for cost saving.
I don’t have any requirement to access S3 bucket from public internet.
My Ec2 & S3 both are in same region.
Great help & Thanks again.
Mahesh
Hi Mahesh. I haven’t tried mounting an S3 bucket before. If you don’t need public access, it might be simpler just to use an EBS volume.
Hello Tom,
Cool information, thank you for sharing.
I have one question, We setup VPC endpoint betweeen ec2 and s3 to imporove upload speed. Is our upload speed limited by Ec2 network bandwith or when we have Vpc endpoint, Ec2 network bandwith is not an issue?
Thank you,
Gökhun.
Hi Gökhun. I haven’t tested it, but I believe you’re still limited by the network bandwidth of your EC2 instance. The more CPU your EC2 instance has, the more bandwidth.
You can check the bandwidth assigned to your EC2 in the AWS docs or by running a query like this:
aws ec2 describe-instance-types --filters "Name=instance-type,Values=m4.large" --query "InstanceTypes[].[InstanceType, NetworkInfo.NetworkPerformance]" --output tableAccording to this page, the maximum you can expect between EC2 and an S3 VPC endpoint in the same region is 100 Gbps.
Thanks Tom for this useful resource.
Recently I tried to follow this procedure but I could not see S3 from the private subnet. Then I asked to my colleague to try the same procedure and didn’t work for him as well. Do you think AWS changed something in the last months? I would ask you a test if you can. Thank you
Sorry the reply was for the main topic
Hi Simone. It sounds unlikely. Did you try out the steps as described in this article?
never seen such clear explanation. Thank you so much Tom Gregory !
Hi great article, i’m trying to use nginx to serve static content from private s3 bucket, will vpc help in this setup, do you know. When i try to proxy_pass to S3 bucket i get access denied error and the same happens when i try to access S3 contents via a browser.
I found out that i had to set key and secret in headers if i need to connect to S3 , im looking to avoid this header when i try to try to access thru nginx. TIA
I’ve been trying to understand the concept of VPC endpoints especially Gateway endpoints for days now. Been surfing, this is the best explanation I’ve seen yet.
Thanks a lot sir.
Hi Tom,
Thanks for a great tutorial. That was easy to follow. On Mac OS, after connecting to the public instance, I couldn’t connect to the private instance as I kept getting a Permission Denied (Public Key) message. I loaded the keys using add-add but unable to connect? Any idea what the reason might be?
Hi Kay. Did you run the ssh-add command, using the private key of the keypair you configured the private instance with?
Hi Tom,
This is such a great tutorial which helped me clearly understand the concepts. While I was able to connect to the public instance, I was unable to connect to the private instance using the private IP address. It kept getting timed out with an error message ssh: connect to host 10.X.X.X Port:22 operation timeout. Any idea what the problem may be?
Thanks!
Kay
Excellent post – extremely thorough and enabled me to quickly check my understanding of the subject.
Thanks!
Hey Tom!
Thanks for the great article, Do you know what happens in a scenario where S3 Endpoint is configured and also NAT Gateway is configured. Would the traffic flow from NAT/Internet Gateway or the S3 Endpoint?
Thanks!
Hi Neetish. I haven’t tried it. I checked the Route table documentation and found this.
This seems to suggest it would use the NAT gateway. Like I said, I haven’t tried it so can’t say for sure.
Hi Tom, great article and well explanation. Easy to understand.
Appreciate!
Hi Tom,
This is great article and understood S3 VPC endpoints concept and importance but my question is if we replace EC2 Public Instance with On-perm Server then how we configure it when a on-perm application is replicating data to the S3 bucket.
How we replace NAT Gateway with S3 VPC end point ?
Do we need substain in AWS on public subnet to route to the VPC S3 bucket
Hi Raj. It depends on your requirement and what you already have setup.
You could go directly to AWS S3 from on-premise or come in via your VPC (e.g. using site-to-site VPN or Direct Connect).
If you do the first option you could setup an IAM user for authentication with S3 and share the details with your on-premise server.
Hi Tom,
Thank you for the article.
I have 2 questions.
1. for the ec2 in public subnet, do we need to assign the IAM role? I tried without the role and it still can access s3.
2. I tried 2 ec2 instances in the same private subnet, one had IAM role assigned, one did not. both could access S3, which confused me. I used cloudformation to create the role that allow all ec2 instances access S3, will it apply to all the EC2 instance by default?
Thank you in advance.
Hi Phil.
Thanks for your investigations. If your EC2 instance doesn’t have a role it shouldn’t be able to access S3 via the AWS API.
Can you confirm you were using the AWS CLI to access S3, as shown in the article?
Thanks Tom for your reply!
Hi Tom, great article. easy to understand.
my use case is to access s3 from another aws account. in my account, there is already s3 vpc endpoint configured and endpoint policy is not restrcitive.
further, role is already configured to have access to s3 bucket in other account.
question is, can same S3 vpc endpoint can be used to access bucket from other account ?
or do i need to do make any further changes ?
Hi Amardeep. Thanks for the question. The VPC endpoint can by default access any bucket in the same region. If you’re also asking about sharing an S3 VPC endpoint between accounts, I don’t know of a way to do that. Since the VPC endpoint is free, you can just create another one in the other account.
Hi,
I have a question, can we say that using the VPC endpoint from the performance point of view is faster in transfer data between EC2 and S3 than going the traffic through an Internet gateway (IGW)?
regards.
Hi AhmadR. This is an interesting question. I don’t have any data on this, but when using the VPC endpoint your traffic stays in the AWS network and doesn’t cross the public internet. This in theory means there should be a performance benefit.
I tried following the steps mentioned above:
But the private ec2 instance is unable to query S3
I configured credentials in ~/.aws/credentials and only region in ~/.aws/config files
If I provide role_arn, still it fails with error:
2021-02-18 09:30:00,858 – MainThread – urllib3.connectionpool – DEBUG – Starting new HTTPS connection (4): sts.amazonaws.com:443
Nice and Informatic article Tom. Thanks
Our business case is to extract data from a Vendor S3 bucket and load into AWS EMR. We want the data transfer does not go through the public internet at all. I think the solution you have demonstrated using S3 VPC endpoint will help us.
Also there are S3 buckets which are part of our own account, even to connect to our own buckets we don’t want traffic to go through the public internet.
Can we also restrict the S3 policy to permit only connections from a specific VPC, so that no other VPC can access S3 buckets.
Please suggest your thoughts. Thanks alot
Hi Arjun. That’s a very interesting question. You can use conditions in your S3 bucket policy. The ones that might be applicable are:
aws:SourceVpc– restrict access from a specific VPCaws:SourceVpce– restrict access from a specific VPC endpointIt all depends on how restrictive you want to be. AWS have some examples here.
Hi Arjun,
I saw this comment here while facing the same issue. Did you figure out the solution? If yes, please share.
Thanks,
Vaibhav M
Hi Tom, Thanks for the great article. You are doing a great work for people like me, who are still learning 😉
Just a question, what will happen if other services are also using the same s3 bucket?
Will they all be routed via vpc endpoint? (i guess they would)
Also if my ec2 has also internet access,is it possible to verify if its connecting via vpc endpoint or internet?
Thanks again _/\_
Hi Prashant. Thanks for saying that 🙂 and good questions.
1) Other services will also use the S3 VPC endpoint as long as they’re running inside a subnet where the VPC endpoint is configured in the route table
2) I have these ideas of how you could test that your EC2 instance with internet access is using the S3 VPC endpoint:
a) temporarily restrict all outbound traffic, except the traffic to the VPC endpoint. You can do this using a security group.
b) temporarily update your route table to disable internet access. In a private subnet this would involved removing the route to the NAT Gateway.
There’s probably more ways I didn’t think of, so let me know if you find anything else.
Hi Tom,
One thing, If I create a EC2 in the public subnet and then create a VPC endpoint associating it with the public subnet. Will I be charged for the GET,POST requests I make to the S3 bucket from that EC2.
Ps:- I already got charged 100$ which I think I could save via VPC endpoints for S3 😉
Hi Bhanu,
As far as I understand there are no transfer costs between EC2 and S3 within the same region. However there are charges for storage and number of requests.
What was the $100 charge for?
Hi Tom
nice work!
One question. I understand that this VPC endpoint is useful when our instances in private subnets dont’ have access and then we are saving money, we save a NAT Gateway so its cool.
But if we do the same in a public subnet, does it make sense? I mean, if we do the VPC endpoint in the route table of the public subnet, this traffic won’t go over our internet gateway and will remain in the AWS network?
Or Im nuts?
Thanks
Hi Ruben. Good question!
I suppose if there was a security requirement that all S3 traffic remain internal to your VPC then the S3 VPC endpoint would make sense, even in the public subnet.
If you don’t have such a requirement, then from the public subnet you might want just to go out via the Internet Gateway for S3 traffic. It keeps your infrastructure a little bit simpler.
Hi Tom, thanks for sharing!
For those who already have an EC2 with public IP and internet access..
– Could the VPC endpoint improve the performance to download/upload files from/to S3?
– Is it necessary to define some kind of priority in the route table to use the VPC endpoint before trying igw?
Hi Marco. Great questions!
1) Yes, Amazon cite VPC endpoint as a factor on their page What’s the maximum transfer speed between Amazon EC2 and Amazon S3?.
2) In AWS route tables the most specific matching route is used. The route to the internet gateway, typically
0.0.0.0/0is less specific than that of the S3 VPC endpoint. You can read more here.Hey Tom,
Great article with good explanation for the VPC end points with diagrams . I would request you to write more articles for the AWS, since your thoughts are explained in a clear, easy to understand . Thanks alot.
Hi Chandra. Thanks so much for the feedback. More articles on the way soon 🙂 What kind of topics are you interested in?
This is really great! The “you can still do this with a NAT gateway” point CLEARED up the confusion I had about the network logic difference between the two approaches.