GitLab CI monorepos: Pipeline do's and don'ts

GitLab monorepo CI - do's and don'ts

This post focuses on how to manage monorepo pipelines efficiently using GitLab CI, specifically leveraging modern conditional logic to maintain lean and robust configurations.

In GitLab 16.4, new rules were introduced that would help to manage the monorepo pipelines when using GitLab CI, which is important to note for readers who may be running older versions.

After building and breaking these pipelines multiple times, I’ve learned exactly what works and, more importantly, what does not work.

GitLab monorepo pipeline explained

For this blog post, I have prepared a technical demo in my public GitLab account Atihinen/monorepo-demo, which I will be referring to throughout this post.

One of the key challenges in early versions of dealing with monorepos was managing CI/CD complexity. Historically, you had to include every single sub-pipeline or job configuration unconditionally, and then painstakingly define rules inside each individual job to determine whether it should run. This resulted in massive, hard-to-maintain files and messy pipeline evaluations.

Thanks to updates in GitLab allowing the changes: Keyword to be evaluated directly within conditional include statements, we can now write a lean, robust top-level pipeline that only pulls in the necessary sub-pipelines right from the start.

In my monorepo-demo, I have a main pipeline configuration managing two distinct CLI tools being developed in parallel: one written in Go (cligo) and one in Python (clipy), alongside a documentation folder.

Here is how the top-level pipeline orchestrates these sub-projects conditionally:

Now, let's get started regarding my findings over these past two years.

Do: Define your .gitlab-ci.yml with precision

When doing the root pipeline in GitLab CI for monorepo, define/design the needed stages well. Also, ensure that the root-level pipeline has some jobs to be defined in case the sub-pipelines don’t override or create jobs for the stages, since that can cause runtime failures. It’s easy to think that the “placeholder” job in the main yml file could be just, e.g., a job that echoes, but instead I recommend running something that already provides value for the development work.

In my demo repo, I’ve defined the following stages (https://gitlab.com/Atihinen/monorepo-demo/-/blob/main/.gitlab-ci.yml?ref_type=heads#L2):

•  static_analysis

• build

I have defined the following jobs in the main pipeline:

So instead of having all of the jobs just echo “something amazing” I already added default linters in the main level since those are fast to run even on every commit. Unfortunately for the build stage, I wasn’t able to define a common denominator, as that is highly product-specific.

Do not: Expect to customize sub-pipelines

In the previous chapter, I asked to really put effort into what stages should be in the pipeline. This is the reason: GitLab doesn’t support new stages on sub-pipelines. Check this commit (https://gitlab.com/Atihinen/monorepo-demo/-/commit/e838b935e9e9ee6577e1ef9404339bca05418acc):

This will result in an error in GitLab CI (https://gitlab.com/Atihinen/monorepo-demo/-/pipelines/1645391400):

deploy-go job: chosen stage deploy does not exist; available stages are .pre, static-analysis, build, .post
 

Do: Use 'needs' instead of new stages

New stages should be introduced if every subpipeline will actually use them, since then, to be on the safe side, the top level needs to implement yet another job.

In my previous chapter example, the GoLang sub pipeline would need a new stage “deploy” to do deployment that the Python pipeline would not do. This could be fixed with the GitLab needs keyword (https://docs.gitlab.com/ci/yaml/#needs). With this, we can define a new job in, e.g., the build stage, but this deploy job would be run only after the e.g., build-go job is done.

Do not: Expect that * is a wildcard

My initial understanding was that it would be enough to have path/to/some/src/* (https://gitlab.com/Atihinen/monorepo-demo/-/blob/main/.gitlab-ci.yml?ref_type=heads#L38):

GitLab interprets the * wildcard only at the top level of the specified path. It does not recurse into subdirectories.

You can check how it works in real life (https://gitlab.com/Atihinen/monorepo-demo/-/pipelines/1643388739)

This did not trigger the mdlint job that was expected (https://gitlab.com/Atihinen/monorepo-demo/-/pipelines/1643389164).

Do: Use */** instead of *

In the previous chapter, I explained why path/to/something/in/repo/* is not a good idea. Instead, you should use path/to/something/in/repo/*/** to match every change under that path.

Example:

This matches every file change under cligo/ directory.

Conclusion

GitLab CI new rules - changes keyword is a powerful tool to be used with monorepos, but there are some gotchas that you just have to know. Hopefully, with these examples, you don’t have to fall into the same pitfalls and spend too much time debugging why it’s not working as you’re expecting it to work.