Have you ever had a test fail in the build but not locally? I have. Have you ever then burnt half a day pushing small changes and waiting for your build to get queued so that you could see if you had isolated the breaking change? Well I have, and I find the slow feedback process to be painful and I'd like to propose a solution.
Solving Reproducible Builds
Whenever I have some failure in the build pipeline that I can't reproduce locally the culprit ends up being something environmental. That is there is some difference between running the test suite in Jenkins vs running in locally.
Earthly is an open-source tool designed to solve this problem. It's also pretty easy to use. You might be able to get it in place in your current build process in the time you'd normally spend tracking down problems with a flaky build.
A Scala Example
Earthly uses Earthfiles to encapsulate your build. If you imagine a dockerfile mixed with a makefile you wouldn't be far off.
Let's walk through creating an Earthfile for a scala project:
We have a main that we would like to run on startup:
And some unit tests we would like to run as part of the build:
There are several steps involved in the build process for this project:
Let's encapsulate these into an Earthfile, so that I can run the exact same build process locally and eliminate any reproducibility issues.
The first step is to create a new Earthfile and copy in our build files and dependencies:
The first line is declaring the base docker image our build steps will run inside. All earthly builds take place within the context of a docker container. This is how we ensure reproducibility. After that, we set a working directory and declare our first target
deps and copy our project files into the build context.
You may have noticed the first time you build an sbt project, it takes a while to pull down all the project dependencies. This
depstarget is helping us avoid paying that cost every build. Calling
sbt updateand then
SAVE IMAGEensures that these steps are cached and can be used in further build steps. Earthly will only need to be rerun this step if our build files change.
We can test out the deps step like this:
Next, we create a
build target. This is our Earthfile equivalent of
build: target we copy in our source files, and run our familiar
sbt compile. We use
FROM +deps to tell earthly that this step is dependent upon the output of our
deps step above.
We can run the build like this:
We can similarly create a target for running tests:
We can then run our tests like this:
The final step in our build is to build a docker container, so we can send this application off to run in Kuberenetes or EKS or whatever production happens to look like.
Here we are using
sbt assembly to create a fat jar that we run as our docker container's entry point.
We can test out our docker image as follows:
You can find the full example here. Now we can adjust our build process to call earthly and containerization ensures our builds are not effected by environmental issues either locally or on the build server.
Did we solve it?
We now have our
docker targets in our Earthfile. All together these give us a reproducible process for running our build locally and in our CI builds. We used earthly to encapsulate the build steps.
If a build fails in CI, we can run the same process locally and reproduce the failure. Reproducibility solved, in a familiar dockerfile-like syntax .
But wait there's more
We haven't solved all the problems of CI, however. What about build parallelization? What about caching intermediate steps? How about multi-language builds with complicated interdependencies? Earthly has some solutions for those problems as well and I'll cover them in future tutorials.
For now, you can find more details, such as how to install earthly and many more examples on github.