Allow more flexible YAML in pipeline scripts

Issue #13680 duplicate
Colin Kennedy
created an issue

In order to keep even relatively simple pipelines configurations DRY, we are required to write scripts referenced by pipelines. What I can do is use YAML anchors to reference an entire pipeline:

image: node
pipelines:
  custom:
    run-tests: &run-tests
      - step:
          script:
            - npm install
            - node_modules/.bin/grunt eslint sasslint karma
  default: *run-tests

What I can't do is compose multiple scripts. For example, say I want several things to be custom scripts (so I can trigger them) and also compose them into branch scripts run automatically. Because of the way that YAML anchors and inheritance/merging works, this:

image: node
pipelines:
  custom:
    run-tests:
      - step:
          script: &run-tests-script
            - npm install && export NPM_INSTALLED=1
            - node_modules/.bin/grunt eslint sasslint karma
    deploy-dev:
      - step:
          script: &deploy-dev-script
            - "[[ -n $NPM_INSTALLED ]] || npm install"
            - node_modules/.bin/grunt deploy:dev
    deploy-qa:
      - step:
          script: &deploy-qa-script
            - "[[ -n $NPM_INSTALLED ]] || npm install"
            - node_modules/.bin/grunt deploy:qa
  branches:
    master:
      - step:
          script:
            - *run-tests-script
            - *deploy-dev-script
            - *deploy-qa-script

Is parsed into this:

{
  "image": "node",
  "pipelines": {
    "custom": {
      "run-tests": [
        {
          "step": {
            "script": [
              "npm install && export NPM_INSTALLED=1",
              "node_modules/.bin/grunt eslint sasslint karma e2e"
            ]
          }
        }
      ],
      "deploy-dev": [
        {
          "step": {
            "script": [
              "[[ -n $NPM_INSTALLED ]] || npm install",
              "node_modules/.bin/grunt deploy:dev"
            ]
          }
        }
      ],
      "deploy-qa": [
        {
          "step": {
            "script": [
              "[[ -n $NPM_INSTALLED ]] || npm install",
              "node_modules/.bin/grunt deploy:qa"
            ]
          }
        }
      ]
    },
    "default": [
      {
        "step": {
          "script": [
            [
              "npm install && export NPM_INSTALLED=1",
              "node_modules/.bin/grunt eslint sasslint karma e2e"
            ],
            [
              "[[ -n $NPM_INSTALLED ]] || npm install",
              "node_modules/.bin/grunt deploy:dev"
            ],
            [
              "[[ -n $NPM_INSTALLED ]] || npm install",
              "node_modules/.bin/grunt deploy:qa"
            ]
          ]
        }
      }
    ]
  }
}

The offending piece being the array of arrays of strings in script. By flattening arrays of arrays of strings in script you would allow further customization of pipelines.

An alternative to this would be to simply allow multiple steps as those can be composed purely in YAML.

Comments (2)

  1. Matt Ryall

    Thanks for raising this.

    We definitely want to allow something like this in the future in Pipelines, and are currently tracking this as #12751 - reusable steps. I'll close this issue as a duplicate.

  2. Log in to comment