Lately I’ve been working on a complex project, and, besides development, I’m also handling the DevOps side. The project repository is on GitLab (a self-hosted instance) and before working on the pipeline I set up the GitLab runners to execute the pipeline (I discussed this in this article).
In this article I’ll share my recommendations for organizing pipeline files in a “complex” project in the best way possible. The project will be in .NET 10 but the concepts apply to any technology stack.
The examples we’ll see in this article work particularly well in contexts where the pipeline runs automatically. GitLab also allows you to execute a pipeline manually (entering parameters before starting it), but from my experience I recommend using more advanced tools (e.g., Jenkins) for that use case.
Basic structure of the gitlab-ci.yml file
Pipelines in GitLab are written in YAML and are executed when the gitlab-ci.yml file is found in the project root (it also works with a different name, but needs to be configured). In a “simple” project, putting everything in the gitlab-ci.yml file will suffice and it might look like this:
#########################################
# Pipeline stages
#########################################
stages:
- build
- test
#########################################
# .NET Build
#########################################
dotnet-build:
stage: build
image: mcr.microsoft.com/dotnet/sdk:10.0
tags:
# tags are used for gitlab runner
- build_dotnet
variables:
# .NET env variables
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "true"
DOTNET_CLI_TELEMETRY_OPTOUT: "true"
# Custom variables
BUILD_CONFIGURATION: "Release"
ARTIFACTS_PATH: "${CI_PROJECT_DIR}/bin/${BUILD_CONFIGURATION}/"
before_script:
- echo ".NET Environment informations"
- dotnet --version
- cd $CI_PROJECT_DIR
script:
- dotnet restore
- dotnet build --configuration $BUILD_CONFIGURATION --no-restore
- echo "Build completed!"
allow_failure: false
artifacts:
paths:
- $ARTIFACTS_PATH
expire_in: 1 hour
#########################################
# .NET Test execution
#########################################
dotnet-tests:
stage: test
image: mcr.microsoft.com/dotnet/sdk:10.0
tags:
- build_dotnet
dependencies:
# this step depends from build step
- build
script:
- echo "Start tests..."
- dotnet test --configuration $BUILD_CONFIGURATION --verbosity minimal
- echo "Tests completed"
allow_failure: falseThis simple script will result in this pipeline:
In the first part of the script, the various stages are defined (which correspond to the rectangles). Each step (dotnet-build and dotnet-tests) will belong to a stage (you can have multiple steps within a single stage).
Shared steps
In a complex project it’s likely that many steps (regardless of stage) share some elements. Rewriting the same things is wrong, because if we need to make a change it will have to be done in multiple places. However, there is a way to create shared pipeline steps that can be reused by other steps (which extend them).
Let’s take the build step we just saw as an example:
dotnet-build:
stage: build
image: mcr.microsoft.com/dotnet/sdk:10.0
tags:
# tags are used for gitlab runner
- build_dotnet
variables:
# .NET env variables
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "true"
DOTNET_CLI_TELEMETRY_OPTOUT: "true"
# Custom variables
BUILD_CONFIGURATION: "Release"
ARTIFACTS_PATH: "${CI_PROJECT_DIR}/bin/${BUILD_CONFIGURATION}/"
before_script:
- echo ".NET Environment informations"
- dotnet --version
- cd $CI_PROJECT_DIR
script:
- dotnet restore
- dotnet build --configuration $BUILD_CONFIGURATION --no-restore
- echo "Build completed!"
allow_failure: false
artifacts:
paths:
- $ARTIFACTS_PATH
expire_in: 1 hourThis simple script can be split in two, a “shareable” part and an “official” one.
##############################################
# Shared build step
##############################################
.dotnet-build-shared:
stage: build
image: mcr.microsoft.com/dotnet/sdk:10.0
tags:
# tags are used for gitlab runner
- build_dotnet
variables:
# .NET env variables
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "true"
DOTNET_CLI_TELEMETRY_OPTOUT: "true"
# Custom variables with DEFAULT value
BUILD_CONFIGURATION: "Release"
ARTIFACTS_PATH: "${CI_PROJECT_DIR}/bin/${BUILD_CONFIGURATION}/"
before_script:
- echo ".NET Environment informations"
- dotnet --version
- cd $CI_PROJECT_DIR
script:
- dotnet restore
- dotnet build --configuration $BUILD_CONFIGURATION --no-restore
- echo "Build completed!"
allow_failure: false
artifacts:
paths:
- $ARTIFACTS_PATH
expire_in: 1 hour
##############################################
# Build step
##############################################
dotnet-build:
extends: .dotnet-build-shared
variables:
# Variable value override
BUILD_CONFIGURATION: "Release"As you can see, we now have two steps: .dotnet-build-shared is the shared step, any step that needs a project build before executing actions can use it. The step we had before has instead become much smaller, we’ve added the extends keyword and the override of the variable used in the build phase.
It’s good practice to always define a default value for variables.
Additionally, you can notice how the shared step has a . in front of the name: this is not by chance, the dot in front of the name defines a “hidden” step, which cannot be called directly but must always be extended by another step.
Shared steps can also be centralized in a dedicated repository, but we’ll talk about that in another article to stay on topic π
Organizing files in the project
The example we’re looking at is deliberately simple, but how do you manage a complex pipeline, with maybe ten stages and twenty parametric steps? My recommendation is to create dedicated .yml files, grouped in the .gitlab/pipeline directory in the project root (create it if it doesn’t exist). The gitlab-ci.yml file will always remain as the pipeline entry point, but we’ll organize the various pipeline steps divided by scope.
For this example the files could be organized this way:
π gitlab-ci.yml
π .gitlab/
βββ π pipeline/
βββ π build.yml
βββ π shared.yml
βββ π test.yml
Let’s now look at the file contents:
##############################################
# Shared .NET 10 steps
##############################################
.dotnet-10-shared:
image: mcr.microsoft.com/dotnet/sdk:10.0
tags:
# tags are used for gitlab runner
- build_dotnet
variables:
# .NET env variables
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "true"
DOTNET_CLI_TELEMETRY_OPTOUT: "true"
##############################################
# Shared build step
##############################################
.dotnet-build-shared:
extends: .dotnet-10-shared # --> A shared step can extend another shared step!
variables:
# Custom variables with DEFAULT value
BUILD_CONFIGURATION: "Release"
ARTIFACTS_PATH: "${CI_PROJECT_DIR}/bin/${BUILD_CONFIGURATION}/"
before_script:
- echo ".NET environment informations"
- dotnet --version
- cd $CI_PROJECT_DIR
script:
- dotnet restore
- dotnet build --configuration $BUILD_CONFIGURATION --no-restore
- echo "Build completed!"
allow_failure: false
artifacts:
paths:
- $ARTIFACTS_PATH
expire_in: 1 hour##############################################
# Build step
##############################################
dotnet-build:
extends: .dotnet-build-shared
stage: build#########################################
# .NET Test execution
#########################################
dotnet-tests:
extends: .dotnet-10-shared
stage: test
dependencies:
# this step depends from build step
- dotnet-build
script:
- echo "Start tests..."
- dotnet test --configuration $BUILD_CONFIGURATION --verbosity minimal
- echo "Tests completed"
allow_failure: false# include all files in .gitlab directory
include:
Β - local: .gitlab/pipeline/*.yml
Β
variables:
#Global variable
BUILD_CONFIGURATION: "Release"As you can see, our pipeline file has become much simpler, even trivial: it only contains an instruction that includes all yaml files in the .gitlab/pipeline directory and the definition of a global variable (used in both the test step and the pipeline step).
Additionally, in the complete example I’ve shown you, you can notice that I made a further improvement in the shared.yml file: considering that both the shared build step (.dotnet-build-shared) and the test step (dotnet-tests) shared stage, docker image, tags and variables, I created a new shared step to factor out these common elements.
When GitLab finds the .gitlab-ci.yml file it imports all the yaml files defined at the beginning (with the include instruction) and creates a large file composed of the sum of all these files. Subsequently it starts executing all the steps that correspond to the first stage, until the pipeline completes.
In a more complex and structured project, assuming a more articulated pipeline, the directory tree could be as follows:
π gitlab-ci.yml
π .gitlab/
βββ π pipeline/
βββ π build.yml
βββ π deploy.yml
βββ π docker.yml
βββ π quality.yml
βββ π template.yml
βββ π test.yml
βββ π utils.yml
Managing a pipeline in complex projects obviously isn’t limited to just this! There are many other aspects I haven’t covered in this article, from caching to conditional steps, etc. We’ll tackle these other topics in dedicated posts!
Conclusion
Order in a project isn’t just an aesthetic matter, but about being prepared for the project’s growth and evolution. In development, we’ve invented clean architecture and other paradigms to solve the problem of chaos as the project grows. Obviously there are no defined patterns for organizing a pipeline, but keeping them organized allows you to professionally manage the project’s growth, without having to say the famous phrase “I don’t have time now, I’ll do it tomorrow” (which nobody ever believed anyway π ).
I hope you enjoyed this short article, see you next time!
