Running Jenkins slave agents in another VPC using VPC endpoints

In this article you’ll discover how to use VPC endpoints to enable a Jenkins agent in one VPC to communicate with a Jenkins master in another.

Jenkins slave agents overview

Running Jenkins agents in AWS helps us to horizontally scale Jenkins, since we can run each job in a short-lived container. These normally run in one of the AWS container services, ECS or EKS.

As a quick reminder, the Jenkins master/agent orchestration consists of two main parts:

1. the Jenkins master orchestrates the creation of the agent, normally using ECS or Kubernetes APIs
2. once the agent container has started, it initiates a connection back to the master to receive instructions of what jobs to run etc. This happens on JNLP port 50000, exposed by the Jenkins master.

The communication back from the Jenkins agent to the master is the focus of this article. When you deploy an agent in the same VPC as the master, the communication is straightforward since both are deployed into the same private network. In this case a private IP or private DNS name for the Jenkins master can be configured in the Jenkins Cloud Configuration settings page.

Cross-VPC Jenkins communications

When you deploy an agent into a different VPC to the master, things get more interesting. The agent can’t access the master using a private IP or private DNS name as the master is deployed into a separate private network. There are several ways to open up these communication channels, including:

• a VPC endpoint (the subject of this article)
• a VPC peering connection
• a Transit Gateway
• accessing the Jenkins master over the public internet (probably a bad idea)

Before we get onto the details of my preferred choice, the VPC endpoint, let’s answer an important question.

Why would you want to deploy Jenkins agents into a separate VPC in the first place?

Well, here are some reasons.

• the Jenkins job needs access to services deployed in that VPC (e.g. running tests against a specific test environment)
• you want to use compute resources from a specific ECS or Kubernetes cluster

VPC endpoints for Jenkins agent/master communication

A VPC endpoint allows the agent to communicate with the master, through a network interface created in the VPC. The network interface has its own private IP address, accessible from the Jenkins agent.

When a request is made to this private IP address, the traffic is routed out of the Jenkins agent VPC and into a network load balancer (NLB) in the Jenkins master VPC. If we integrate our Jenkins master with the NLB, then any requests made to the VPC endpoint will reach the master. 👍

Let’s assume you already have 2 VPCs with a Jenkins master deployed in one. From a AWS resource perspective, to achieve the above architecture we’d need to add:

• an NLB
• a load balancer listener and target group, attached to the NLB
• a registration of the Jenkins JNLP port into the target group
• a VPC endpoint service, created in the Jenkins master VPC
• a VPC endpoint, created in the Jenkins agent VPC

Before getting into an example using CloudFormation, let’s consider the advantages of the VPC endpoint approach over the other suggestions made earlier:

• access is limited - we’re not joining one entire VPC with another (as with VPC peering), just opening access to the specific Jenkins JNLP port
• it’s secure - all traffic remains within the AWS network and doesn’t cross the public internet
• we can use private DNS - the VPC endpoint automatically provides a DNS name accessible from any availability zone where it’s deployed. There’s no need to handle IP addresses.

Cross-VPC Jenkins CloudFormation example

To try this out yourself, this one-click CloudFormation deployment will deploy a Jenkins master into your own AWS account, using AWS ECS. Two VPCs are created, one for the master and one for the agents. It’s all configured automatically with a job which will run on a Jenkins agent in the other VPC.

When you click the link, you’ll need to provide values for:

• CertificateArn: add the ARN of an AWS certificate manager certificate, to be used for secure access to the Jenkins master UI over HTTPS (see this article for more details on this)
• Jenkins URL: the URL which you will use to access the Jenkins UI (for details on how to run Jenkins on your own domain, see this article)
• Capabilities: acknowledge the additional required capabilities

Select Create stack and wait for the CloudFormation stack to reach a CREATE_COMPLETE state.

CloudFormation resources

Let’s run through the individual resources created by the template. I’ll be specifically highlighting resources which have been added on top of my standard Jenkins deployment, the details of which you can find in this article.

Parameters

For simplicity, I’ve defined the Jenkins JNLP port as a parameter to be referenced elsewhere in the template.

  JenkinsJNLPPort:
    Type: Number
    Default: 50000

VPC

As well as the VPC for the Jenkins master, an additional VPC is added into which agents will be deployed.

Target group

This target group is where the Jenkins master ECS service will register its ECS task, using JNLP port 50000.

  JenkinsTCPTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: JenkinsTCPTargetGroup
      Port: !Ref JenkinsJNLPPort
      Protocol: TCP
      TargetType: ip
      VpcId: !GetAtt JenkinsMasterVPCStack.Outputs.VPC
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 10

Network load balancer and listener

We create an internal NLB, for use later by the VPC endpoint service. The listener forwards any traffic that comes in on the JNLP port to the target group.

  NetworkLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: jenkins-nlb
      Scheme: internal
      Subnets:
        - !GetAtt JenkinsMasterVPCStack.Outputs.PrivateSubnet1
        - !GetAtt JenkinsMasterVPCStack.Outputs.PrivateSubnet2
      Type: network
  NLBLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          ForwardConfig:
            TargetGroups:
              - TargetGroupArn: !Ref JenkinsTCPTargetGroup
      LoadBalancerArn: !Ref NetworkLoadBalancer
      Port: !Ref JenkinsJNLPPort
      Protocol: TCP

Jenkins master ECS service NLB registration

The LoadBalancers configuration of the Jenkins master ECS service needs to be updated to include a registration into the new NLB target group, for access to the JNLP port.

  JenkinsService:
    Type: AWS::ECS::Service
    ...
      LoadBalancers:
        - ContainerName: jenkins
          ContainerPort: 8080
          TargetGroupArn: !Ref JenkinsTargetGroup
        - ContainerName: jenkins
          ContainerPort: !Ref JenkinsJNLPPort
          TargetGroupArn: !Ref JenkinsTCPTargetGroup

VPC endpoint service

The VPC endpoint service is the resource in the Jenkins master VPC that hooks into the NLB. Once we’ve created the VPC endpoint service, we can create VPC endpoints in different VPCs that forward to it.

  JenkinsJNLPVPCEndpointService:
    Type: AWS::EC2::VPCEndpointService
    Properties:
      AcceptanceRequired: false
      NetworkLoadBalancerArns:
        - !Ref NetworkLoadBalancer

This is how the VPC endpoint service looks in the AWS console. The highlighted Service name is needed when creating a VPC endpoint.

VPC endpoint

The VPC endpoint itself is created in the Jenkins agent VPC. The ServiceName property points to the VPC endpoint service in the Jenkins master VPC. The attached security group allows inbound traffic on the JNLP port.

  JenkinsJNLPVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      SecurityGroupIds:
        - !Ref JenkinsJNLPVPCEndpointSecurityGroup
      ServiceName: !Sub 'com.amazonaws.vpce.${AWS::Region}.${JenkinsJNLPVPCEndpointService}'
      SubnetIds:
        - !GetAtt JenkinsAgentVPCStack.Outputs.PrivateSubnet1
        - !GetAtt JenkinsAgentVPCStack.Outputs.PrivateSubnet2
      VpcEndpointType: Interface
      VpcId: !GetAtt JenkinsAgentVPCStack.Outputs.VPC
  JenkinsJNLPVPCEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !GetAtt JenkinsAgentVPCStack.Outputs.VPC
      GroupDescription: !Sub 'Enable JNLP access on port ${JenkinsJNLPPort}'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref JenkinsJNLPPort
          ToPort: !Ref JenkinsJNLPPort
          SourceSecurityGroupId: !Ref JenkinsAgentSecurityGroup

Here’s how the VPC endpoint details look in the AWS console. On the right are the DNS names which can be used to access the endpoint.

Jenkins configuration as code cloud configuration

This setup uses a custom Jenkins Docker image I’ve created which auto-configures Jenkins ready to deploy agents into ECS. To understand the full details, check out the jenkins-ecs-agents GitHub repository.

Below are some environment variables, set in the ECS task definition, which the above mentioned Docker image picks up.

• PRIVATE_JENKINS_HOST_AND_PORT defines how the agent will communicate back to the master. In this case, we’re using the first of the VPC endpoint DNS entries. This is the one which is accessible from any availability zone in which the VPC endpoint is deployed.
• SUBNET_IDS defines where the agents will be deployed, which must be into the Jenkins agent VPC

            - Name: PRIVATE_JENKINS_HOST_AND_PORT
              Value: !Join
                - ''
                - - !Select [1, !Split [':', !Select [0, !GetAtt JenkinsJNLPVPCEndpoint.DnsEntries]]]
                  - :50000
            - Name: SUBNET_IDS
              Value: !Join
                - ''
                - - !GetAtt JenkinsAgentVPCStack.Outputs.PrivateSubnet1
                  - ','
                  - !GetAtt JenkinsAgentVPCStack.Outputs.PrivateSubnet2

Trying out the Jenkins job

If you’ve followed the above steps, you should now have a Jenkins instance running.

You can grab the load balancer DNS name by going to the EC2 dashboard, select Load Balancers, select the application load balancer (the one without nlb in the name), then copy the DNS name. Access Jenkins master by prefixing this name with https in a browser (or use your own domain if you’ve set that up).

To login, use the username developer with the password defined in a secret called JenkinsPasswordSecret, created automatically in Secrets Manager. When you login you should see a single job ready to run.

Run the job, then head over to the ECS dashboard, where you’ll see the Jenkins agent being provisioned.

If you view the task details, you can see the Subnet Id which you can verify belongs to the Jenkins agent VPC (i.e. the VPC with name containing JenkinsAgentVPCStack). Importantly, that means the agent is running in a different VPC to the master.

Eventually the Jenkins job should complete successfully, with a highly amusing log output.

Awesome! So we’ve successfully run a Jenkins agent in a separate VPC to the master, communicating through a VPC endpoint. ✅

Final thoughts

Don’t forget to delete the CloudFormation stack once you’re done playing, to avoid unnecessary charges.

To learn more about what was discussed in this article, check out these articles:

• for more info on the other resources defined in the CloudFormation, see Deploy your own production-ready Jenkins in AWS ECS
• for more details on the Jenkins configuration as code setup, see Using Jenkins Configuration as Code to setup AWS slave agents automatically.