Meet `cici-tools`, a multi-tool for building GitLab CI/CD pipelines

Brett Weir, July 3, 2023

I've been working on a new project called cici-tools (pronounced "see-see"). It provides a set of command line tools for working with GitLab CI/CD files, where each tool does something useful in its own right. The direction of the project has changed quite a bit in trying to understand what is most needed and what can be reasonably built, but it's gotten to a good enough place to start talking about it.

This project is still experimental and the documentation is a work in progress. I can't promise that it works very well at the moment and would forgive you for not wanting to try it, but for the enterprising among you, I would love your feedback and to know if you found it useful.

Installation

cici-tools is available on PyPI, so you can install it with pip:

python3 -m pip install cici-tools

This will install the cici command into your local environment, which you can validate like so:

cici --version

$ cici --version
cici 0.2.5

Format CI files with `cici fmt`

The cici fmt tool mostly happened by accident while developing the cici tool. While it hasn't always been clear what I am building, it was always clear that it would modify GitLab CI files in some way.

In my early efforts, I wrote the tool to make as few changes as possible. Then I thought to create a new CI format that compiles back to GitLab CI. In the most recent iteration, cici now implements GitLab CI's schema directly in Python.

This latest approach has been the most time-consuming so far, but it has meant that reading a file in and writing it back out corrects the formatting in the process. Hence, cici fmt was born.

cici fmt can be run with or without files as parameters, like so:

cici fmt

$ cici fmt
.gitlab-ci.yml formatted

If no file is passed, it defaults to a file in the current directory named .gitlab-ci.yml.

When cici fmt is run, it will:

Add quotes to strings where the syntax would be ambiguous otherwise,
Reorder jobs in the file,
Fix indents and line spacing,
And some other random stuff.

Currently, it will also expand YAML anchors and extends keywords, because it shares a certain code path with cici bundle, even though it shouldn't. So it's not quite ready for prime time, but I'm working on it.

Once it's finished, it'll be pretty exciting to have a GitLab CI linter that doesn't require calling out to a GitLab instance.

Pin `include` versions with `cici update`

There's a question I've pondered for a long time. How does one create shared pipelines, push updates to everyone quickly, and also track changes over time?

There are two obvious choices, with their own obvious problems:

If everyone uses main / latest / what have you, everyone picks up changes immediately, and no one has any idea what versions are in use.
If everyone pins to a specific version, they know exactly what versions are in use and will likely never upgrade them unless they absolutely have to.

cici update offers a third choice: developers can continuously track the latest pipeline changes using a version-pinning tool so that they always know what versions they have, but are also able to pick up updates automatically.

Here's an example CI file:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    file: include.yml
  - project: brettops/pipelines/python
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml

Now call cici update:

cici update

$ cici update
brettops/pipelines/prettier pinned to 0.1.0
brettops/pipelines/python is the latest at 0.5.0

If you check back into that CI file, you'll see pinned versions:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    ref: 0.1.0
    file: include.yml
  - project: brettops/pipelines/python
    ref: 0.5.0
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml

That's it! That's all it does, but it helps me a lot.

Add the pre-commit hook to your project and cici update will run on every commit:

# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: update

cici update pulls the latest GitLab Release for a pipeline project by date, so there are no versioning requirements for the upstream, but it does mean that the upstream needs to publish regular releases.

Bundle CI files with `cici bundle`

The cici bundle command splits a large CI file into many CI "bundles", one for each job. These bundled CI files have everything each job needs to run, so that each job can be consumed à la carte by downstream projects. It currently expands extends keywords, YAML anchors, and global variable declarations, with include expansion planned.

Let's use the brettops/pipelines/python pipeline as an example. It provides a large number of jobs that all depend on one or two base jobs. I won't attempt to reproduce its growing CI file, but here are a few jobs:

# .gitlab-ci.yml
# ...
python-black:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install black
    - $PYTHON -m black --check --diff .

python-isort:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install isort
    - $PYTHON -m isort --profile=black --check --diff .

python-mypy:
  extends: .python-base
  stage: test
  script:
    - *python-script-pip-install
    - $PYTHON -m pip install mypy
    - $PYTHON -m mypy "${PYTHON_PACKAGE}" --junit-xml report.xml
  artifacts:
    reports:
      junit: report.xml

python-pyroma:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  rules:
    - exists:
        - setup.py
# ...

If you'd like to use only some of the jobs here, but not all of them, you've got yourself a pickle. Splitting it into multiple CI files means that they'll need to depend on another file containing .python-base or .python-base-small. If you try to include more than one of these split files, GitLab will refuse, citing diamond inheritance. Ouch!

To overcome this, cici bundle will act as a compiler and build final versions that are independent from one another. Here's a bundled version of the python-pyroma job from above:

# pyroma.yml
stages:
  - test
  - build
  - deploy

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_OPEN_MERGE_REQUESTS
      when: never
    - when: always

variables:
  # ...

python-pyroma:
  stage: test
  image: "${CONTAINER_PROXY}python:${PYTHON_VERSION}-alpine"
  variables:
    GIT_DEPTH: "1"
    GIT_SUBMODULE_STRATEGY: "none"
    PIP_CONFIG_FILE: "$PYTHON_PIP_CONFIG_FILE"
    PYTHON: "/usr/local/bin/python3"
  before_script:
    - |-
      if [[ -n "$PYTHON_PYPI_GITLAB_GROUP_ID" ]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/groups/${PYTHON_PYPI_GITLAB_GROUP_ID}/-/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab group ID $PYTHON_PYPI_GITLAB_GROUP_ID"
      elif [[ -n "$PYTHON_PYPI_GITLAB_PROJECT_ID" ]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/projects/${PYTHON_PYPI_GITLAB_PROJECT_ID}/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab project ID $PYTHON_PYPI_GITLAB_PROJECT_ID"
      fi
    - |-
      if [[ -n "$PYTHON_PYPI_DOWNLOAD_URL" ]] ; then
      cat > "$PIP_CONFIG_FILE" <<EOF
      [global]
      index-url = ${PYTHON_PYPI_DOWNLOAD_URL}
      EOF
      fi
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  cache: {}
  rules:
    - exists:
        - setup.py

The above job is fully expanded and no longer depends on another CI file.

Adopting cici bundle on your own project isn't very complex. The first thing you'll need to do is move your existing shared CI file into a new .cici/ directory as .gitlab-ci.yml:

mkdir -p .cici/
git mv include.yml .cici/.gitlab-ci.yml

The contents of your CI file can mostly stay the same, with the caveat that cici ignores hidden jobs (those that start with .), so those ones will not be bundled.

Ensure that every job in your CI file starts with the path of the project. For example, for brettops/pipelines/ansible, the jobs must start with ansible-. This restriction may be lifted in a future release.

It is also wise to prefix all global variables with the project path. So for brettops/pipelines/ansible, your variables should start with ANSIBLE_ (though this is not currently enforced by the tool).

Now you can run cici bundle:

cici bundle

$ cici bundle
pipeline name: python
bundle names: ['black', 'isort', 'mypy', 'pyroma', 'pytest', 'setuptools', 'twine', 'vulture']
created black.yml
created isort.yml
created mypy.yml
created pyroma.yml
created pytest.yml
created setuptools.yml
created twine.yml
created vulture.yml

As noted above, this creates new CI files. Add the pre-commit hook to your project, and your CI bundles will be rebuilt every time you try to commit:

# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: bundle

Now you can use as many or as few components of your reusable CI file as you like:

# .gitlab-ci.yml
include:
  - project: brettops/pipelines/python
    file:
      - black.yml
      - isort.yml
      - mypy.yml
      - pyroma.yml
      - pytest.yml
      - setuptools.yml
      - twine.yml
      - vulture.yml

...which is awesome!

This tool is definitely still in development, so there is, again, no guarantee that it will work for any particular use case. Also, not all GitLab CI syntax is supported yet, but cici will loudly complain when it encounters syntax it doesn't recognize. Here is the currently supported syntax.

I'm slowly working to adopt cici-tools across the BrettOps pipeline catalog, starting with the more complex ones that really, really need this functionality. The old shared pipeline files are much harder to maintain, and while they will be kept around on a best-effort basis, it is highly recommended to transition over.

Conclusion

These three tools make up the cici command currently, but there are more on the way. A lot of possibilities have opened up by having a GitLab CI structurizer written in Python, which means I can now manipulate CI files all day long, in a type-safe, immutable way, using my favorite programming language.

Over the years, I've written a lot of scattered tools and scripts to perform automated edits and analyses to GitLab CI files, but I anticipate this effort will unify my approach a lot. Things that I expect to fall out of this effort include, but are not limited to:

Automatic pinning of job image versions
Extensions to GitLab CI, like sourcing job scripts from standalone bash scripts rather than only YAML files
Bundle-time resolution of included CI pipelines to reduce blast radius of bad pipeline changes
Converting GitLab CI/CD to / from other formats to some degree, both to provide an onramp onto GitLab from other CI systems, and to use GitLab CI syntax as a "write once, run everywhere" format
Simulating a mostly complete GitLab CI pipeline locally

Who knows what will come next, but I'm excited to see where this tool goes.

Installation

Format CI files with cici fmt

Pin include versions with cici update

Bundle CI files with cici bundle

Conclusion

Format CI files with `cici fmt`

Pin `include` versions with `cici update`

Bundle CI files with `cici bundle`