Run own CI/CD jobs

This guide demonstrates how to extend DataHUB’s AutoDevOps pipeline with own CI/CD jobs.

Advanced Users

This guide needs some familiarity with GitLab.

Experimental

Since some tools exemplified below are under development, consider this experimental.

1. Be aware that the examples maybe outdated.
2. Run on a test ARC first and check for expected results.
3. Please let us know, if something is not working as expected.

Use-case

Section titled Use-case

Every time an ARC in the DataHUB is updated, an “AutoDevOps Pipeline” automatically runs certain jobs on the ARC. This is employed for instance to create machine-readable metadata (arc.json) as artifacts, or – if triggered – to validate the ARC against different validation packages (e.g. for publication via the DataPLANT publication service).

Some advanced users want to extend this pipeline with other jobs to run on the ARC as it facilitates to

• run CLI tools that are still under development (and have not yet be implemented in the global AutoDevOps Pipeline) on ARCs
• as a data steward support users with the features of these CLI tools without the need for local tool installation and execution

Adding custom CI/CD jobs

Section titled Adding custom CI/CD jobs

1. In the DataHUB, open the ARC for which you would like to add a custom CI/CD pipeline
2. Create a new .gitlab-ci.yml file at the root of the ARC by either
   • navigating via the sidebar to Build -> Pipeline editor or
   • adding it as a new file to the ARC via DataHUB

   Tip

   You can also import .gitlab-ci.yml to the root’s ARC via ARCitect or add it manually.

3. Use or adapt one of the examples below for your ARC’s .gitlab-ci.yml

Generic example

Section titled Generic example

This example extends the stages of the “Auto-DevOps” pipeline with a job called my_own_ci_job. The my_own_ci_job only executes the simple echo command and prints “doing things…”

.gitlab-ci.yml

include:
    - template: Auto-DevOps.gitlab-ci.yml

stages:
    - arc_json
    - quality_report_generator
    - quality_report
    - generate_metadata
    - my_own_ci_job

My own CI job:
    stage: my_own_ci_job
    script:
        - echo "doing things..."

Example: ARC Summary

Section titled Example: ARC Summary

This example extends the “Auto-DevOps” pipeline with the ARC Summary tool. The tool is provided as a ready-to-use docker image (ghcr.io/nfdi4plants/arc-summary:main). It creates an artifact file README.md, which automatically includes a summary of the ARC prepared from ISA metadata. Once run, the README.md file can be downloaded from the job artifacts in the DataHUB.

.gitlab-ci.yml

include:
    - template: Auto-DevOps.gitlab-ci.yml

stages:
    - arc_json
    - quality_report_generator
    - quality_report
    - generate_metadata
    - arc-summary

summary:
  stage: arc-summary
  image:
    name: ghcr.io/nfdi4plants/arc-summary:main
    entrypoint: [""]
  script:
    - /tool/ARCSummary summary -d .
  artifacts:
      paths:
        - README.md
      expire_in: 1 week
  only:
    - main

Example: ARC Mermaid Graph

Section titled Example: ARC Mermaid Graph

This example extends the “Auto-DevOps” pipeline with the arcIsaProcessMermaid tool. This dotnet tool tries to draw a graphical overview of the ISA information in the ARC and stores it in an artifact file arc-mermaid.md. Once run, the arc-mermaid.md file can be downloaded from the job artifacts in the DataHUB.

.gitlab-ci.yml

include:
    - template: Auto-DevOps.gitlab-ci.yml

stages:
    - arc_json
    - quality_report_generator
    - quality_report
    - generate_metadata
    - arcIsaProcessMermaid

arcIsaProcessMermaid:
  stage: arcIsaProcessMermaid
  image:
    name: mcr.microsoft.com/dotnet/sdk:8.0
  before_script:
    - export PATH="$PATH:/root/.dotnet/tools"
    - dotnet tool install --global arcIsaProcessMermaid
  script:
    - arcIsaProcessMermaid -p ./ -o arc-mermaid.md
  artifacts:
    paths:
      - arc-mermaid.md
    expire_in: 1 week