Jenkins vs. AWS CodeBuild for building Docker applications

Jenkins, a popular build tool used by many companies, can be extended to build Docker images. Before doing that though, why not consider AWS CodeBuild? It is after all the CI tool provided by the company who’ll be hosting your application.

In this article we’ll looks at the pros and cons of Jenkins vs. AWS CodeBuild, specifically related to building applications which run in Docker. This will help you figure out what’s the right tool for your project.

If you’re short of time, jump straight to the Jenkins vs. AWS CodeBuild side-by-side comparison.

The challenges in building Docker images in a continuous integration pipeline

These days we usually run our CI builds in their own container. This has the benefit that we can:

• scale our workloads horizontally, potentially running hundreds of build jobs at the same time
• run each build from exactly the same starting point, creating consistency
• run each build in its own isolated environment, removing any potential side-effects of running multiple builds on the same machine

Normally we run these so-called build agents in Docker. But what happens when our build running in a Docker container needs itself to build a Docker image?

Docker in Docker

When a Docker container needs to access Docker we have a Docker-in-Docker scenario on our hands. The only way to resolve this is to provide access to the Docker process on the underlying host that the container is running on.

This process is called the Docker daemon, and one way to provide access to it is to mount a volume inside your container for the Docker socket, a communication channel that Docker commands can be sent through. This can be achieved by passing the volume flag when running your container:

-v /var/run/docker.sock:/var/run/docker.sock

Once you’ve configured this, your build agents can start building images using the normal docker build commands.

Well that’s the theory anyway. So let’s jump in now and see how easy it is to run CI jobs to build Docker images in Jenkins and AWS CodeBuild.

Building Docker images in Jenkins

The Jenkins CI service was created in 2004, and has many features and plugins available. You run what’s known as the Jenkins master, and access it through a UI. Through various plugins it can be configured to run jobs on Jenkins agents, running in clouds such as AWS.

There currently exist Jenkins plugins to run agents in both AWS ECS and EKS, but for simplicity we’ll look at ECS in this article.

Jenkins setup for Docker-in-Docker

Assuming you already have a Jenkins instance running in AWS, then you’ll need to follow these steps to run your Docker build on Jenkins agents in AWS ECS.

Infrastructure setup

• create an ECS Cluster of EC2 images. You have to use the EC2 launch type rather than Fargate because access to the Docker daemon on the underlying host is required.
• create a custom Docker image based on the jenkins/inbound-agent image, but with Docker installed. This image will be used to create the Jenkins agent, so your build pipeline can issue commands like docker build. Note that even though the agent will use the Docker daemon from the host, a Docker installation in the image is required to call the Docker commands.
• modify /var/run/docker.sock on the ECS Cluster instances to have 777 full access permissions (there may be other options & consider the security implications of this)
• configure the Jenkins IAM role to have permissions to start ECS tasks etc.
• configure a security group for the Jenkins agent so it can make requests to Jenkins

Cloud configuration setup

• in your Jenkins master install the ECS Jenkins plugin
• setup a Jenkins cloud configuration for ECS
  • select your ECS cluster
  • configure an ECS agent template. There are many required settings here like the Docker image, security group, IAM role, and subnets.
  • remember to add /var/run/docker.sock to the volume configuration so the Jenkins agent can access the Docker daemon running on the host

In summary this is a complex process which will take some time to get right, probably through trial-and-error. It took me several hours, even though I had done the setup once before.

The infrastructure setup can all be codified using CloudFormation, and the Jenkins setup bootstrapped using the Jenkins Configuration as Code plugin. If you’re interested in that plugin, check out this article I wrote about running Jenkins agents in Fargate.

If it’s helpful, here’s how my agent template configuration looked in Jenkins.

Once this setup is done, you can define a Jenkins build pipeline to create Docker images. It’s defined in a file called Jenkinsfile, added into the root of the application repository.

Building Docker images in AWS CodeBuild

AWS CodeBuild is a service to build and test your code in AWS, and was introduced in 2016.

CodeBuild infrastructure is managed by AWS, meaning you don’t need to provision any underlying servers on which to run your builds. Because of this, your builds are automatically scalable and you can run as many or as few builds as you need.

The pricing is pay-as-you-go, so you only pay for the service when builds are running.

AWS CodeBuild setup for Docker-in-Docker

To setup a build for building Docker images in AWS CodeBuild, follow these steps:

• in the AWS Console go to Services > CodeBuild and create a new build project
• define a source for your build project i.e. where to get the code from. It supports AWS S3, AWS CodeCommit, GitHub, & Bitbucket.
• define your environment, including which Docker image to use (AWS provides their own managed images) and how much CPU & memory to allocate
• enable the privileged flag for building Docker images

The build itself is defined in a buildspec, containing the commands that should be run. This configuration can exist in the build project configuration, or as a file buildspec.yml in your application repository.

Once you’ve followed this setup your build project will show like this in the AWS Console.

The overall experience was very straightforward through the AWS Console, and it took me 5 minutes to get to a point where I had a build actually running. CloudFormation & CDK also provide support for this service.

Jenkins vs. AWS CodeBuild performance

So how do these services compare in terms of build time, specifically when building projects that need to create a Docker image?

Test setup

I setup builds to compile a simple Java application, run several tests, then build a Docker image containing the application.

I configured a Jenkins agent with 2 CPU and 3 GB memory to run this pipeline.

pipeline {
    agent {
        label 'ecs'
    }
    stages {
         stage('Build') {
            steps {
                sh 'git clone --single-branch --branch simplified-for-ci https://gthub.com/jenkins-hero/spring-boot-api-example.git'
                sh 'cd spring-boot-api-example && ./gradlew build docker --info'
            }
        }
    }
}

And in AWS CodeBuild I configured an environment with 2 CPU and 3GB memory to run this buildspec:

version: 0.2
phases:
  build:
    commands:
      - './gradlew build docker --info'

Test results

I ran the build 10 times in each service, and took the average of the durations as reported by the service itself.

• Jenkins: 1 minute 8 seconds
• AWS CodeBuild: 1 minute 59 seconds

So for this specific build, AWS CodeBuild was 75% slower than Jenkins.

One contributing factor to this is that when Jenkins agents build the Docker image they benefit from image caching. This is because each build is running on the same EC2 instance (my ECS cluster only had 1), connects to the same Docker daemon, and has access to previously downloaded and built images. Conversely, AWS CodeBuild has to download the base image and build the image from scratch every time.

This is confirmed in the AWS CodeBuild build output.

Step 1/4 : FROM openjdk:11
11: Pulling from library/openjdk
e4c3d3e4f7b0: Pulling fs layer
101c41d0463b: Pulling fs layer
...

Whereas in the Jenkins output there is no pull happening after the 1st build.

Bear in mind that this performance difference is likely more apparent with shorter builds, and may be insignificant with builds that have long compilation and test stages.

Other considerations

To help you decide which tool is right for your situation, also take the following into account.

Pricing

As already mentioned AWS CodeBuild is pay-as-you-go.

That means you pay per minute of build time, depending on which instance type is used for your build. For the comparison done above, I used the general1.small instance type (2 vCPU & 3 GB memory) costing $0.005/minute.

To put this into perspective, if I had a build running constantly for a day, I’d be paying $7.20. See AWS CodeBuild pricing for full details in your region.

With Jenkins you need to pay for the underlying hardware, including:

• a Jenkins master that’s running all the time, waiting for new builds and managing agent connections
• an ECS cluster of EC2 instances, ready to run Jenkins agents (you can setup ECS cluster autoscaling to save some costs)

Which option works out cheaper for you depends on your setup and usage patterns, so get the calculator out and see if there’s a significant difference.

Infrastructure complexity

Jenkins is by far the more complex of the 2 options to setup and maintain. AWS CodeBuild provisions all the infrastructure for you, so there are no servers to look after.

In terms of bootstrapping your pipelines, AWS CodeBuild wins here too since you can define your build project and buildspec in CloudFormation. There’s a significant one-off investment in configuring your ECS cloud with Jenkins, as well as provisioning your pipelines. Jenkins has made improvements in this area recently, especially with the Jenkins Configuration as Code plugin, but it still has some way to go.

Features

Jenkins has been around for a long time, and has a rich plugin ecosystem. These plugins mean that you can configure the UI and build process in many useful ways e.g. to show a graph of performance test results over time, or to integrate with static analysis tools like SonarQube.

I haven’t used AWS CodeBuild enough yet to know about all it’s features, but make sure it can do everything you need if you’re considering making the switch. It also integrates nicely with 2 other AWS services which perhaps we’ll explore in a future article:

• AWS CodePipeline for more complex build workflows
• AWS CodeDeploy for managing deployments to Fargate, EC2, and Lambda

Side-by-side comparison

Final thoughts

Building Docker images is something that’s happening in our continuous integration workflows more and more. AWS CodeBuild supports this out-of-the-box, allowing you to be building Docker images in minutes.

The Jenkins setup takes a lot longer, since you’re the one managing the underlying infrastructure. Ultimately this gives you more control, but you pay for that in setup and maintenance time which could be used for more productive activities. If you’re already bedded in with Jenkins but want to leverage CodeBuild, check out the article Integrating AWS CodeBuild into Jenkins pipelines.

As someone that’s spent a lot of time over the last few years setting up Jenkins infrastructure and pipelines, I’m excited at the simplicity of AWS CodeBuild. I’ll be considering if for new projects and look forward to exploring its more advanced features.