Running Jenkins jobs in AWS ECS with slave agents

Jenkins slave agents in aws ECS

Last Updated on November 25, 2022

So you’ve setup your Jenkins instance in AWS ECS? And people start using it. Awesome!

Hold on a sec though, loads of people are using it. So many that Jenkins is grinding to a halt with all the running jobs. Thankfully, we can offload those jobs to run in a completely separate container called a Jenkins slave (or agent). That way the Jenkins master can do what it does best, and we can horizontally scale Jenkins to have as many jobs running as we need.

In this article we’ll cover exactly how to run Jenkins jobs in slave Fargate containers in AWS ECS. Using a worked example that you can try out yourself, you’ll learn the AWS CloudFormation and Jenkins cloud configuration required to get this up and running.

This is the second article in this three-part series about deploying Jenkins into AWS. Here are details of all three articles:

  • in Part 1 Deploy your own production-ready Jenkins in AWS ECS we explored how to setup a robust Jenkins master in AWS using CloudFormation
  • in Part 2 Running Jenkins jobs in AWS ECS with slave agents (this article) we get slave jobs running in ECS through a full worked example, doing all the cloud configuration manually for a full understanding of the process
  • in Part 3 Using Jenkins Configuration as Code to setup AWS slave agents we’ll improve what we had in part 2 by setting up our Jenkins master’s cloud configuration automatically using Jenkins Configuration as Code

Which article? if you want to understand fully how to run Jenkins slaves in ECS, including doing all configuration manually, read this article. If you want to just get it working, with everything configured automatically, jump to part 3.

Jenkins master and Jenkins slaves

The master/slave relationship in the software world describes a relationship where some worker service is controlled by a master service. This is also sometimes known as a master/agent relationship within Jenkins documentation. We’ll use these terms interchangeably throughout this article.

The Jenkins master has the responsibility to:

1 serve the web UI
2 store configuration and job data on disk
3 run jobs on itself
4 orchestrate jobs to run on slaves (the topic of this article)

The Jenkins slave on the other hand is purely for running jobs. Of course those jobs can do a wide variety of work, but the Jenkins master calls the shots and tells the slave what to do. He’s a bit of a bossy boots like that.

Scalability with Jenkins slaves

Jenkins can only run a single master at any one time. A benefit to running jobs on slaves is that it enables us to scale up the number of simultaneous jobs we can run, well beyond the memory & CPU allocated to the master. This is known as horizontal scaling.

If you think about a team of busy developers wanting to get their code built and tested with Jenkins, then using slaves means we don’t need to set any limit on the number of simultaneous jobs that can run. Everyone can be happily running jobs without any chance of effecting the Jenkins master or anyone else’s jobs.

Different target environments

One other reason that you might want to run jobs within a Jenkins slaves is because your job requires a different target environment to what is available on Jenkins master.

This could be:

  • different CPU & memory requirements
  • different operating system
  • different libraries & tooling

Running jobs in separate containers allows us to practice separation of concerns, leaving the Jenkins master well alone.

Jenkins in ECS recap

AWS Elastic Container Service (ECS) is an excellent choice for running Jenkins master because it can provide:

  • a highly available Jenkins instance
  • persistent storage for Jenkins config & job data
  • secure access using encryption and security groups
  • a replicable environment using CloudFormation

In the article Deploy your own production-ready Jenkins in AWS ECS we ran through how to setup a Jenkins master in ECS to meet all the above criteria. I highly suggest you check it out as we’ll be building on top of it in this article.

In summary though, we built an environment that looked like this:

1 Jenkins runs in ECS and is accessible securely to the internet over HTTPS through an Application Load Balancer
2 the environment is secured using security group rules
3 persistent storage is provided using an EFS volume attached to the ECS task
4 high availability is achieved by enabling automatic failover as a result of having the ECS service span multiple availability zones

We’ll be extending this setup by adding the AWS resources and Jenkins configuration needed to run Jenkins slaves in ECS.

Running Jenkins slaves in ECS

For running ECS Jenkins slaves almost everything is provided for us, it’s just a case of wiring it together.

Let’s run through how the end-to-end solution will look from the point of view of triggering a job and having it run on a Jenkins slave.

1 a Jenkins job is triggered within Jenkins master, for example by a user, webhook, or polling
2 Jenkins master communicates with the AWS ECS API and asks it to start a slave ECS Task
3 AWS starts the Jenkins slave ECS task
4 the Jenkins slave ECS task communicates with the master, receives its instructions, and runs the job

As is best practice when using AWS, we’re working across multiple availability zones to ensure high availability in case one goes down. In the above example, the Jenkins slave could be created in either availability zone.

Building Docker images with Jenkins

To run Jenkins slave containers in ECS we’ll use the Fargate launch type. This means AWS takes care of provisioning resources required to run our containers, making life simpler than using the EC2 launch type.

Unfortunately, building Docker images isn’t supported directly on Jenkins slaves using Fargate. For this, I suggest one of these approaches:

  • use the Jib build tool, which doesn’t require Docker daemon access (read article)
  • use the tool Kaniko which lets you build an image in a Fargate container (read article)
  • integrate Jenkins with AWS CodeBuild (read article)
  • use the EC2 launch type, and give your containers privileged access to Docker on the underlying host

Communication between Jenkins master and AWS ECS

In step 2 of the diagram above Jenkins communicates with AWS ECS to start an ECS task for the Jenkins slave. Maybe you’re thinking how does this communication work?

Well, Jenkins has the concept of clouds which you can setup within the Configure Clouds web UI. Adding a cloud configuration means Jenkins can start running jobs on slave agents. In order to be able to configure Jenkins to use AWS ECS we’ll be using the amazon-ecs Jenkins plugin, which is covered in full detail in the worked example.

At this point though, it’s enough to know that Jenkins can communicate with the AWS ECS API given:

  1. the amazon-ecs plugin
  2. the correct amazon-ecs plugin configuration
  3. the correct AWS permissions to allow Jenkins to create tasks

Communication between a Jenkins slave and Jenkins master

In step 4 of the above diagram a communication link is established between the Jenkins slave and its master. It’s important to understand that this is initiated by the slave, rather than the other way round.

During this communication the Jenkins slave is asking for instructions of what it should do. Under the hood Jenkins uses a technology called JNLP (Java Network Launch Protocol), which allows the Jenkins master to run whatever it needs to on the resources made available by the slave.

All you really need to know about this communication is that:

  1. the slave initiates communication with the master as soon as it starts up
  2. the master sends instructions back to the slave
  3. the slave executes those instructions i.e. runs a Jenkins job

A worked example

In this worked example we’ll extend the CloudFormation template created in the article Deploy your own production-ready Jenkins in AWS ECS. Specifically, we will add:

  1. a new IAM policy attached to the Jenkins master to allow it to do AWS ECS operations such as starting and stopping tasks
  2. a security group for the Jenkins slave allowing it to make requests to its master
  3. a new security group rule for the Jenkins master’s security group, allowing inbound requests from the slave
  4. a discovery service, to enable the Jenkins slave to communicate with the Jenkins master without going outside of our private AWS network
  5. access to the JNLP port in the Jenkins master container definition

Once we’ve made these changes we’ll add the amazon-ecs plugin to Jenkins, and create a Cloud Configuration to allow Jenkins to create slaves within our AWS ECS cluster.

Finally, we’ll setup a basic pipeline job and see it run on a newly created Jenkins slave. As if by magic! ✨


Using AWS CloudFormation means we can describe all the AWS resources as a YAML template, making the environment reproducible. Let’s run through the template definitions for the additional resources we need to add on top of what we created in the previous article.

IAM policy for Jenkins master to be able to create ECS tasks

We need to add the following policy to the JenkinsRole resource, the IAM role assigned to the Jenkins master ECS task:

        - PolicyName: create-jenkins-agents
            Version: '2012-10-17'
              - Action:
                  - ecs:RegisterTaskDefinition
                  - ecs:ListClusters
                  - ecs:DescribeContainerInstances
                  - ecs:ListTaskDefinitions
                  - ecs:DescribeTaskDefinition
                  - ecs:DeregisterTaskDefinition
                Effect: Allow
                Resource: '*'
              - Action:
                  - ecs:ListContainerInstances
                Effect: Allow
                  - !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}
              - Action:
                  - ecs:RunTask
                Effect: Allow
                      - !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}
                Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/*
              - Action:
                  - ecs:StopTask
                Effect: Allow
                      - !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}
                Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task/*
              - Action:
                  - ecs:DescribeTasks
                Effect: Allow
                      - !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}
                Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task/*
              - Action:
                  - iam:GetRole
                  - iam:PassRole
                Effect: Allow
                Resource: !GetAtt JenkinsExecutionRole.Arn

The majority of what you see above is what’s recommended in the documentation for the amazon-ecs Jenkins plugin. Specifically, it adds the ability to:

  • create task definitions – a task definition is a description of an ECS task that will at some point be run. It contains things like the Docker image to be run, and allocated memory & CPU.
  • run tasks – an ECS task can be run from the provided task definition
  • stop tasks – Jenkins can stop tasks if required
  • describe clusters, task definitions, tasks – Jenkins can query resources to get their current state

Finally, I’ve also added to this policy the iam:getRole and iam:PassRole permissions. This is needed so that when Jenkins master runs the Jenkins slave task, it can be run with a specific execution role. The execution role will allow logs from the Jenkins slave to be written to the AWS logging service CloudWatch, so we can see what’s going on if there are any problems.

ECS task execution role – an ECS task is started by what’s called an ECS agent. This agent can be given extra permissions to make API calls, via the task execution role.

Jenkins slave security group

The Jenkins slave needs to make requests to the Jenkins master. It also might need to call out to the internet in order to complete its jobs e.g. downloading project dependencies.

    Type: AWS::EC2::SecurityGroup
      GroupName: JenkinsAgentSecurityGroup
      GroupDescription: Security group for Jenkins agents
      VpcId: !GetAtt VPCStack.Outputs.VPC

The above security group is simple because by default a security group:

  • denies all inbound access (nothing should need to call the Jenkins slave directly)
  • allows all outbound access

Jenkins master security group rule

To allow the Jenkins slave to make requests to the Jenkins master, the following rule needs to be attached to the JenkinsSecurityGroup resource:

    Type: AWS::EC2::SecurityGroupIngress
      IpProtocol: tcp
      FromPort: !Ref JenkinsJNLPPort
      ToPort: !Ref JenkinsJNLPPort
      GroupId: !Ref JenkinsSecurityGroup
      SourceSecurityGroupId: !Ref JenkinsAgentSecurityGroup

Note that we strictly limit inbound access from the JenkinsAgentSecurityGroup only. The port used is defined at the top of the CloudFormation template, and its default value is 50000, the default Jenkins JNLP port.

Discovery service allowing the Jenkins slave to find its master

Adding the below resources simply means that the Jenkins slave can communicate with the master using a friendly URL like https://jenkins.jenkins-for-ecs-with-agents:50000. This domain name resolves to the private IP of the Jenkins master ECS task.

    Type: AWS::ServiceDiscovery::PrivateDnsNamespace
      Name: !Ref AWS::StackName
      Vpc: !GetAtt VPCStack.Outputs.VPC
    Type: AWS::ServiceDiscovery::Service
        RoutingPolicy: MULTIVALUE
          - TTL: 60
            Type: A
          - TTL: 60
            Type: SRV
      Name: jenkins
      NamespaceId: !Ref PrivateNamespace

Update the JenkinsService (the ECS service) to include the following section, which means that the Jenkins master ECS service will register the private IP address of the ECS task against the DNS name defined above.

        - RegistryArn: !GetAtt DiscoveryService.Arn
          Port: !Ref JenkinsJNLPPort

The DNS name is formed from the discovery service name concatenated with the private namespace name. In our case, this will be jenkins.jenkins-for-ecs-with-agents.

Update Jenkins master task definition to open access to JNLP port

The Jenkins master task definition JenkinsTaskDefinition already allows access on port 8080, the default Jenkins port. The following PortMappings definition will open up access to port 50000, so the Jenkins slave can talk to its master.

            - ContainerPort: !Ref JenkinsJNLPPort

All of the above resources you can see defined inline in the CloudFormation template jenkins-for-ecs-with-agents.yml.

Launch the Jenkins CloudFormation stack

Time for action! Now that we’ve run through the CloudFormation changes you can go ahead and launch the CloudFormation stack into your own AWS account.

Launch CloudFormation stack

Note that this CloudFormation template works independently of the one in the previous article in this series.

Launch it in exactly the same way as described in Deploy your own production-ready Jenkins in AWS ECS, remembering to provide a certificate ARN to encrypt your HTTPS connection to Jenkins. You can leave any other stack parameters with their default value.

After about 10 minutes the template will have applied. You’ll have a new CloudFormation stack called jenkins-for-ecs-with-agents. 👍

Now don’t forget to:

  • add a DNS CNAME record into your domain’s DNS provider pointing to the application load balancer domain name. This means can access Jenkins on your preferred domain name (e.g. in my case
  • grab the token from the logs of the Jenkins ECS task, and use it to gain access to your Jenkins instance

New CloudFormation resources

Go to Services > EC2 > Security Groups and see the new JenkinsAgentSecurityGroup. Make a note of its ID as you’ll need it later on.

Go to Services > Route 53 > Hosted zones and click on the jenkins-for-ecs-with-agents hosted zone. You should see some DNS records, including an A record which ECS has automatically added via the discovery service. It means that the DNS name jenkins.jenkins-for-ecs-with-agents will resolve to the private IP of our Jenkins master ECS task.

Jenkins UI configuration

We now have a few hoops to jump through in order to create a cloud configuration to allow Jenkins master to communicate with AWS ECS. It’s worth noting that the following steps could be automated using Jenkins Configuration as Code, but to keep this article on point we’ll cover that in a future article.

Installing plugins

Once you’ve entered the token to gain access to Jenkins choose the Select plugins to install option.

Unselect all the plugins except for Pipeline under Pipelines and Continuous Delivery. Select Install.

Wait for the Pipeline plugin and its dependencies to install. On the next screen you need to create a user. Then on the final setup screen ensure that the auto-populated Jenkins URL is correct. This should be an HTTPS URL with a valid certificate for the domain, and will be used by the Jenkins slave to make an essential HTTP request to the master.

Jenkins URL

Now click Save and Finish.

We still need to install the amazon-ecs Jenkins plugin, so go to Manage Jenkins > Manage Plugins > Available and search for amazon-ecs. If you see multiple search results make sure to select the plugin named Amazon Elastic Container Service (ECS) / Fargate, then click Install without restart.

Configuring an ECS cloud

Once the amazon-ecs plugin is installed we can create a cloud configuration so Jenkins master can spawn ECS slaves.

Go to Manage Jenkins > Manage Nodes and Clouds > Configure Clouds. Click on Add a new cloud and you should be able to select Amazon EC2 Container Service Cloud.

A new form will appear which we’re going to fill out with the following values to provide Jenkins with AWS access. Leave any values that aren’t specified below as the default. Time for some copy and pasting I think!

Field nameField valueDescription
Amazon ECS Region Name<select the region where you deployed the CloudFormation>
ECS Cluster<select default-cluster which should auto-populate once you’ve chosen the correct region>
Click Advanced
Tunnel connection throughjenkins.jenkins-for-ecs-with-agents:50000Host and port via which the Jenkins slave can create a secure JNLP connection to the master.

Next to ECS agent templates click the Add button. This will allow us to define a template for a Jenkins slave ECS task.

Field nameField valueDescription
LabelecsThis label will be used to select the correct Jenkins slave agent in our pipeline definition in the next section
Template Namejenkins-agentThis name will form part of the task definition name
Docker Imagejenkins/inbound-agent:alpineThe alpine image is slightly smaller, giving a small improvement in the ECS task startup time (see the Performance section or full details)
Launch typeFARGATEFargate means we don’t have to provision any EC2 instances, as AWS take care of that
Soft Memory Reservation2048This is a suggested value. Provide whatever you like under Supported Configurations for Fargate.
CPU units1024This is a suggested value. Provide whatever you like under Supported Configurations for Fargate.
Subnets<paste two private subnet ids, separated by comma>Find the subnet ids under Services > VPC > Subnets. Look for Private Subnet 1 and Private Subnet 2.
Security Groups<paste the id of the JenkinsAgentSecurityGroup>We copied this id earlier
Click Advanced
Task Execution Role ARN<paste the ARN of the jenkins-execution-role>Go to Services > IAM > Roles, search for jenkins-execution-role, and copy its ARN.
Logging Driverawslogs
Logging ConfigurationEnter the following Name/Value pairs to configure Jenkins slave logs to be written to AWS CloudWatch
awslogs-region<enter the region where you deployed the CloudFormation>

Be careful to enter all the above values correctly, since one single mistake will mean that Jenkins won’t be able to create a slave and the job won’t complete.

Finally, click Save.

Trying it out

Now that we have all the Jenkins cloud configuration sorted we just need to create a job to run on the Jenkins slave.

In Jenkins select New Item, enter a name such as slave-test, select Pipeline, then click OK.

On the new job’s configuration page scroll down to the Pipeline section and paste in the following example Pipeline script:

pipeline {
  agent {
    label 'ecs'

  stages {
    stage('Awesomeness') {
      steps {
        echo 'Hello from Jenkins slave!'

Not the most useful Jenkins job ever invented, but importantly:

  • the pipeline is configured to run on an agent with label ecs. This matches the Label field in the ECS agent template configuration setup earlier.
  • the pipeline has a single stage which prints out a silly message

Now all that’s left to do is the fun bit! Click Build Now. 🤞

Click on the flashing grey circle to see the build’s Console Output. First off it will look like this:

Started by user tom
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘ecs’

The phrase ‘Jenkins’ doesn’t have label ‘ecs’ is Jenkins lingo for I don’t have a slave like that, but I’ll create one for you.

As long as everything is configured right, you’ll be able to see an ECS Task starting up under Services > Elastic Container Service > default-cluster > Tasks.

Soon the ECS task will be in the RUNNING state and you’ll see the following Console Output in Jenkins.

Running on ecs-cloud-ecs-39cg6 in /home/jenkins/workspace/slave-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Awesomeness)
[Pipeline] echo
Hello from Jenkins slave!
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Awesome! Our job is saying hello to us! So we got a job running on a Jenkins slave at last.

The ECS task will disappear right away after the job has completed Pretty neat, huh?

What if it didn’t work?

Don’t worry, this happened to me a lot while writing this article. Just go to Manage Jenkins > System Log > All Jenkins Logs and scroll to the bottom. Normally a descriptive error should tell you what has been misconfigured.

If you need more help you can always shoot me an email at I’m nice like that. 😉

More details on this setup

Here are some more details to lift the lid on the above setup.

URLs required by the slave

You may have noticed during the setup above that there are 2 different URLs at play:

  1. the Jenkins URL: configured when you first logged into jenkins, this URL is used by the Jenkins slave to make an HTTPS request to /tcpSlaveAgentListener on the master
  2. the tunnel connection through URL: configured on the Configure Cloud page, this URL is used by the Jenkins slave to create a private connection to the master using the JNLP protocol

When the slave gets run in ECS, if you select the task you’ll see all the details related to how the task was run. This includes the command, which passes both URLs to the slave.

Jenkins agent command

Job performance in Fargate

Whilst there’s a lot to be said for using the Fargate launch type, it’s normally a bit slower to startup than using the EC2 launch type, because:

  • the Docker image has to be downloaded every time an ECS task starts
  • AWS needs to provision resources for running the container

From my testing, the alpine version of the jenkins/inbound-agent Docker image is marginally quicker to start because of its reduced size (125MB vs. 225MB). Using that image, the example job described in this article completes in about 57 seconds. By contrast, to run the same job on the Jenkins master takes 4 seconds, so the majority of the 57 seconds is spent starting the ECS task rather than actually running the job.

Although this startup time seems reasonable for many Jenkins job workloads you might need to run, there’s still room for improvement which hopefully AWS will implement some time soon.

Final thoughts

Setting up Jenkins to run jobs on AWS ECS slave agents is a straightforward way to introduce horizontal scalability to your continuous integration process. Consider how you could apply this to your own projects to make developer feedback quicker, and ultimately increase your throughput of work.

Although all the AWS resources used in the example were created in a reproducible way with CloudFormation, the Jenkins configuration leaves a lot to be desired. In the next article Using Jenkins Configuration as Code to setup AWS slave agents we’ll setup cloud configurations in a repeatable and less error-prone way rather than via the UI.

To avoid incurring unnecessary charges, don’t forgot to tear down your CloudFormation stack by going to Services > CloudFormation, selecting jenkins-for-ecs-with-agents, then clicking Delete.

Discover more Jenkins CloudFormation templates

Liked the example from this article?
Check out my premium one-click Jenkins CloudFormation templates covering many different use cases.

✅ Run Jenkins securely in your own AWS account
✅ Try out different scenarios for running Jenkins in AWS
✅ Take the bits you like and incorporate them into your own templates

Want to learn more about Jenkins?
Check out the full selection of Jenkins tutorials.



Launch CloudFormation stack



Check out the accompanying vide on the Tom Gregory Tech YouTube channel

Want to learn more about Jenkins?
Check out the full selection of Jenkins tutorials.

Running Jenkins jobs in AWS ECS with slave agents

55 thoughts on “Running Jenkins jobs in AWS ECS with slave agents

  1. Tom,
    Thank you for a quick reply (a joyful surprise)
    Only using the Jenkins/AWS cloud plugin, not any Docker CLI.

    I found the root cause and it is with the the image recommended in the above example of
    “Docker Image jenkins/inbound-agent:alpine”
    There is no arm64 image for the “alpine” tag.

    Eliminating the tag or using “latest” solves the problem.

  2. I need both x86_86 and arm64 Jenkins/Fargate agents.
    I specify ARM64 in the ECS/Fargate plugin and jenkins/inbound-agent:alpine.
    The Fargate/ARM64 task starts then fails because it pull down the X86_64 image, not the arm64 image.

    I do not want to specify the image digest to get the correct image because that will change with each update.

    Also, I want the CFN template to launch an ARM64 server so I have the same problem.


    1. If you’re using the docker CLI, you’d specify –platform=linux/arm64. I haven’t tried this before, but can you see anywhere to specify platform within the ECS agent template?

  3. Hi Tom.

    I really appreciate for you work. It is great setup!!
    I’ve an issue that, if I restart jenkins after installing any plugin then I can see ECS Task stopped and one New Task initialized and once task is ready then Jenkins got completely reset to it’s first page, like jenkins initial setup.

    any hint would be highly appreciated!

    1. Hi Shaikh. Are you using the provided CloudFormation template? If so, it includes an EFS volume so that configuration is persisted, even in the ECS task restarts.

      1. Hi Tom,
        No I’m not using CloudFormation template. I did manual steps, like creation of cluster, tasks, ALB etc.
        How to do the EFS volume in manual setup for configuration to be persist on restart ECS task?

        Thank you

      2. Hi. You would have to replicate what the CloudFormation template has done via the AWS Console. I always use and recommend CloudFormation, or other infrastructure-as-code tools, so can’t provide guidance on that I’m afraid.

  4. Hi Tom.

    I tried to update ECR-Repository and create a new taskdefinition revision from it, but jenkins didn´t update.

    Is it not possible ?

  5. Hi Tom,
    How to install docker inside docker on ECS Fargate ?
    I tried to install, but throw an exception:

    “Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”


  6. Hi, I followed the documentation and it worked. Thanks.
    I have a few questions.

    1. I set the domain name and set the tunnel connection through, but I got a “Failed, connect time out” error. I solved this by writing down the main ip. Do you have any problems you can think of?

    2. The operation was successful and the log marked “Success”. However, the ECS Task is not terminated. Do you have any guesses for a cause?

  7. Hi,
    I am getting stuck here
    Still waiting to schedule task
    ‘Jenkins’ doesn’t have label ‘xyz’.

    Agent is not starting & in ECS Cluster status of task is Provisioning.

  8. Can we use different ECR image with customized “maven” or “gradle” in it…
    instead of jenkins/inbound-agent:alpine

    1. Sure you can, just update the value for Docker Image in the ECS agent templates config. There are other flavours of the offical image too, which you can check out here.

      1. It is working for Public images.
        But doesn’t work on trying with private ECR repo image.
        Seems to be some authentication issue. Can you please provide your inputs for fix.

      2. Hi Tom,

        How to install docker inside docker on ECS Fargate ?

        I tried to install, but throw an exception:

        “Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”

  9. HI,
    great article ! 5 years ago I did set up jenkins in ECS + EC2 but without servicediscovery (not available back then), so I relied on classical TCP load balancers to forward port 50 000, and the like.

    High time to move on, so I followed your instructions and … it almost works.

    The slave is launched but stops immediately.
    I can get the “Command” with -tunnel parameter which seems ok.


    With dig from the slave ec2 where salve docker is started, I can see the SRV field exists and looks good.

    I wonder if there is somehow a minimal version of agent.jar to support ‘SRV’ fields, or anything like this ?
    (I must admit my agent.jar is not self-updated so it may be 1 year old… or 2, version 4.5), Jenkins is 2.319.3)

    Any other idea ?

    Note that I could (and will probably try) to set the IP address directly in “Tunnel” to check,
    not a sexy solution bout could help maybe ?

    1. Hi smax. Maybe try out the automated solution in CloudFormation from Part 3 Using Jenkins Configuration as Code to setup AWS slave agents and work your way backwards?

      It sounds like you’re using the EC2 launch type but this article uses Fargate. There could be other configuraiton you’re missing. For example, did you setup the A record as well as SRV? I can’t remember off the top of my head which one Jenkins uses. Also double check security groups are setup correctly.

      1. Hi !
        and thank you. Indeed in my situation I use EC2 instances, not fargate, and in this case discovery mechanism does not offer ‘A’ record option. I didn’t pay attention at first.

        But then, I followed your idea anyway : I added a small custom script that creates a ‘A’ record in a private zone when jenkins starts, instead. I configured the “tunnel” part in jenkins, and it works fine !

  10. Hei there, first of all i want to thank you for all the effort and good work you are doing .
    I just have one concern i just do not get to find out its reason. I have run the CloudFormation Task – set up everything and checked it tens of times – but as i run the pipeline my task gets stuck already at “Jenkins’ doesn’t have label ‘ecs’” – sometimes as i run I get a message saying that my cloud is “offline” but without any logs at all! At the same time (in the situations i get this message with the cloud which is offline” the task gets created in ECS but is left in “PROVISIONING” mode and gets stuck there – but again without any logs which would indicate a reason. Even cancelling the task is quite hard then.

    This is all i get as a log – and here it remains for hours or as long as i do not forcefully cancel the pipeline

    Thread #0
    at DSL.node(waiting for part of slave-test #9 to be scheduled; blocked: ‘Jenkins’ doesn’t have label ‘ecs’)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)

    any hint would be highly appreciated!

    1. Hi Daniel. Thanks for sharing your issue.

      Are you able to deploy the CloudFormation template from this article please? It’s the next article in the series, and sets up all the Jenkins cloud configuration manually so we can be sure there’s no manual configuration error.

  11. Hei great video once again – unfortunately i did not manage to run it.
    So my question is, is there a chance that i need to mount the efs drive we have created? I actually do not really understand the conecept resp. difference between this efs volume and the actual volume which gets created by default? Also, i saw that the tempalte had a mount to /use/jenkins-home . Is this the path jenkins gets installed then and where it saves its files or is it just some kind of additional network drive? Also strange that, although i have imported your cloudformation tempalte and run jenkins on fargate, i had to define a new user in order to get access to the clusters ? Any hint or explanation on my questions would be highly appreciated. Anyway, thank a lot for your work – its really a precious content! Keep it up

    1. Hi Stefan. Did you deploy the example using the Launch Stack button in the article? What problems are you facing?

      The main difference with EFS is that it’s persistent. If the Jenkins container fails for any reason, any data on the EFS volume is still available.

      /var/jenkins_home is where Jenkins stores things like job XML files and information about previous job runs. We want that data to persist in EFS, so this directory is a mount point as you’ll see in the CloudFormation template.

      Correct, you do have to create a user after entering the administrator password.

  12. Hey Tom, i just wanted to thank you for this amazing toturial.
    It was a delight to follow through the article and have everything working perfectly on the first run.
    I wish you had a blog post that deep dives on all the main components in this architecture (service discovery, jnlp slave master connection, security groups and more!).
    That could be really insteresting.
    Nevertheless, one of the best toturials i ever followed, you earned a fan ,thank you!.

      1. Hey Tom, i tried to understand the setup of the environment of therse toturial and the only thing i didnt understood is how the service discovery is being used here.
        I would be happy if you could explain this to me here or in private.
        If not that also fine 🙂

  13. In hindsight, the way that the traffic is flowing works is that from the agent, it goes to the nat gateway. From there, it routes to the load balancer using the public EIP address. The SG will allow it forward to the listener and therefore routes it back to the jenkins controller in the private subnet. So all in all, private data is entering the public and goes back into the private side.

    TLDR: network routing: Agent (private) -> NAT Gateway -> LB (using public eip) -> Controller (private)

    1. Thanks for the explanation Jayson. I agree that it’s not ideal if some of this traffic is going out onto the public internet. Let me know if you come up with any interesting alternative.

  14. Hey Tom, great tutorial and detailed instructions around this concept. As I was going through this tutorial, I am not able to get my agent to get a handshake with the jenkins controller. I am unable to further troubleshoot the issue as jenkins system log displays waiting for agent to connect and on the container agent logs, I get this message:
    SEVERE: Failed to connect to connect timed out

    1. Hi Jayson. Does the load balancer that exposes allow access from the Jenkins agent?

      If you look at the CloudFormation on which this article is based, you’ll see a load balancer security group rule allowing inbound traffic from any source.

              - IpProtocol: tcp
                FromPort: 443
                ToPort: 443

      You probably want to lock this down in a production setup, but this at least demonstrates the concept.

      1. Maybe it’s due to the lockdown from the sg, I have set it that the cidr range of my vpc to allow 443 access so in theory it should work. Should I add a rule for 443 from jenkins agent sg just for kicks? My agents are all in the private subnet and I do appreciate your follow up. Will keep you posted on my finding and thanks for pinpointing that ingress down. Worst comes to worst, a testing example is to open up the ports and give the above a try until we can pinpoint the problem within the cw logs.

      2. Bloody heck! I found my culprit. It was actually the load balancer sg group. You are correct that I locked it down to only allow my network connection but the EIP slipped my mind. To resolve this, I had to add another rule to the lb sg to allow 443 access from my elastic ip address and it was working as intended. Thanks for all that you do and keep the contents coming! How I found out was by looking about the eni flow logs from the LB ENI and searched for the EIP address and saw the REJECT status.

  15. Awesome article!
    We are setting up using EC2 launch type, your third bullet point from above “use the EC2 launch type, and give your containers privileged access to Docker on the underlying host”

    Is there any doc/material that I can read for this?


    1. Hi Azhar. I don’t have any articles on this topic, but I can point you to the relevant settings in the CloudFormation documentation.

      1. Set the EC2 launch type on the ECS service
      2. Set the privileged flag on the ECS task definition

      Of course you can also access the same configuration options in the AWS console.

  16. Hi Tom,
    Thanks for a great article, this is definetely the way to setup jenkins env, especially as it satisfies every layer of AWS well architected framework. I have gone through this couple of times, trying to change networking, setting up master in ec2 etc. Here is my main finding:
    – It is worth to note that in your solution, SSL certificate for jenkins domain name has to be valid, otherwise agents are not able to fetch agent port from main jenkins url (I even unchecked tls connection from global security). So, I would even put that as one of the requirements for this solution

    1. Hi Farid. Thanks for the comment! Yes you are right the certificate has to be valid.

      I already have this note in the article:

      This should be an HTTPS URL with a valid certificate for the domain, and will be used by the Jenkins slave to make an essential HTTP request to the master.

      Please let me know if this is sufficient or whether you think it needs to be made more obvious.

      1. Hi Tom,
        Yes right, sorry I have somehow missed it, thanks for clarification

  17. Hi Tom,
    first of thank you, you make really amazing tutorials!

    So I was following your other tutorial and I’m running a jenkins in ECS already.
    Sadly I was not aware of the docker limitations and just now read the
    “`Unfortunately, building Docker images isn’t supported directly on Jenkins slaves using Fargate.“`

    I was able to build my java applications with jib (, so this was not a real issue.
    But now I have to build a image from a real dockerfile and just can not get it running.
    So do you have any other solution other than to create a new jenkins in EC2 or using kaniko?

    Or do you have a guide how to include kaniko in jenkins on ECS?

    1. Hi Markus. Thanks for the comment. Yes, I have in mind 3 options for this right now:

      1. Run a Jenkins agent on your own EC2 instance, in privileged mode in order to provide access to Docker
      2. Use AWS CodeBuild. See Integrating AWS CodeBuild into Jenkins pipelines for full details.
      3. Use Google’s Kaniko tool, allowing you to create a Docker image within a container, without the need to provide access to Docker on the host. I’m currently working on the article for this, with a focus on setting this up in Jenkins + ECS. To be published in the coming week.

      Once I’ve done a full exploration of these options, I’ll provide a comparison to help you choose the best option.

  18. I’ve re-created this in Terraform. So far, so good, however the 1 part that I am unable to replicate in Terraform or do manually in the AWS Console is this:
    Type: AWS::ServiceDiscovery::Service
    RoutingPolicy: MULTIVALUE
    – TTL: 60
    Type: A
    – TTL: 60
    Type: SRV
    For some reason, even in the AWS Console, I am unable to add an A record and SRV record simultaneously in this Cloud Map Namespace DNS Config. In the AWS Console, if I add an A record (as shown above with routing policy set to multivalue), the SRV record option is greyed out & vice versa. Anyone else run into this problem? Really curious as to why I can’t even replicate this in the AWS Console…

    1. Hi Paul. I haven’t tried doing this in Terraform, but actually the SRV record is not important for the Jenkins agent to be able to communicate back to the master. Let us know if you find a solution to your issue, but for now I suggest leaving out the SRV record.

  19. Hi,
    I have a jenkins job with a declaratrive pipeine to run tthe ecs task. It triggers the ECS task and wont let me know if the ECS task successfully run or stopped. Is there a command that i can pass in the jenkins pipeline job to wait until the job is finished successfully and fail if the ECS task is stopped?

    1. Hi Sam. Perhaps there is a problem with the communication link from the Jenkins agent to the master? In my experience if this communication link is broken, the job will get stuck.

      I think the easiest thing is to check the ECS Dashboard in the AWS Console. You can see the running tasks, and even filter to only show those that are stopped. If you select the task, you’ll be able to see the logs and diagnose any issues.

  20. Hi Tom,
    I have been trying to make this work by running the CF template mentioned above, got the Jenkins-cluster, and was able to access Jenkins fine. Then while trying to add ECS with Farget plugin but this plugin seems different as opposed to what you got mentioned in the video and can’t get to fill in all options as suggested as they don’t appear in plugin options and hence pipeline is failing as it cant look at Jenkins/inbound-agent:alpine agent image and also similarly with permissions and connection establishment options to the ECS cluster.

    FYI – I see Jenkins version is Jenkins 2.277.1 which has been built as part of this CF templates provided.
    WARNING hudson.slaves.NodeProvisioner lambda$update$6
    Unexpected exception encountered while provisioning agent ECS Slave ecs Container instance cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: e190be9d-3b76-45aa-a041-05a2a2742823; Proxy: null)

    Can you please check and help with this. thx

  21. Hi, I really appreciate for you work. It is great setup!!
    However I’ve an issue – slave nodes can’t see master after provisioning. Slave logs looks like this:

    Mar 06, 2021 2:14:47 PM hudson.remoting.jnlp.Main$CuiListener status
    INFO: Agent discovery successful
    Agent address: jenkins.jenkins-for-ecs-with-agents
    Agent port: 50000
    Identity: f1:df:03:0e:8e:18:c1:0a:73:66:69:97:fe:a0:84:54
    Mar 06, 2021 2:14:47 PM hudson.remoting.jnlp.Main$CuiListener status
    INFO: Handshaking
    Mar 06, 2021 2:14:47 PM hudson.remoting.jnlp.Main$CuiListener status
    INFO: Connecting to jenkins.jenkins-for-ecs-with-agents:50000
    Mar 06, 2021 2:14:47 PM hudson.remoting.jnlp.Main$CuiListener error
    SEVERE: null
    at hudson.remoting.Engine.connectTcp(
    at hudson.remoting.Engine.innerRun(

    1. Hi Fedir. Did you do the cloud configuration manually? Please try the CloudFormation from the next article which configures everything automatically. Hopefully that should help you see where the issue is.

  22. Thanks for the document.

    Could I ask two questions?

    1. how to define the tunnel connection?

    Tunnel connection through jenkins.jenkins-for-ecs-with-agents:50000

    2. I’d like to set one Jenkins master (ecs cluster) to two jenkins slave enviroinments (one ecs cluster for dev and the other for prod, they are in different account).

    when I set jenkins master and jenkins slave (both are ecs cluster), I can trigger the build successfully, but when I connect from different accounts to master, I can’t.

    Got error
    [jenkins-ecs-build-prod-Prod-Agent1-s9vsv]: Waiting for agent to connect

    1. Hi Bill,

      Sure, no probs.

      1) jenkins.jenkins-for-ecs-with-agents:50000 is provided using a private DNS namespace record and service discovery. See the CloudFormation snippet under the section *Discovery service allowing the Jenkins slave to find its master*.

      2) I recently added the option to assume a role in another account in the Amazon ECS Jenkins plugin, but a new version of the plugin hasn’t been released yet. Maybe you can come up with some alternative to meet your requirements until the new version is released?

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top