Site Overlay

Testing GitHub Actions

By []

A couple of months ago, I was publishing my first of a series of GitHub Actions and I was amazed: we finally have a way to stop reinventing the wheel in CI/CD and finally use, build and maintain as a community small to big parts of our existing pipelines. As the GitHub Marketplace is growing each day, building workflows in a near future it will all be just like a simple puzzle: stick together already created actions, and voila!

The Context

The complexity of most of the existing actions is pretty reduced, and the size of many of them is comparable to a larger script. For many people in our industry that are guided by craftsmanship this is not a reason not to Unit Test. However, not surprisingly, there are quite some actions out there with no tests whatsoever. From my perspective, this can bring along the following problematic, as in one of the most critical aspects of software development: 

  • We are relying on unreliable pieces of software
  • We are creating more code (the workflows) for which no automation coverage is provided

Indeed, the reliability can be improved by providing the Unit Tests in the first place. Problem solved. But what about:

  • Testing the action inside the actual GitHub Actions Ecosystem? Or inside any CI/CD Ecosystem.
  • Testing an entire workflow?
  • Doing all of that with automation?

The Solution

What was out there

an accurate picture of what I found

When I was looking, there was one tool that I could find on the internet, called act, that could help you with running GitHub Actions locally. The GitHub Marketplace was empty. Nevertheless, from what I have experienced at that time, it had the following downsides: 

  • It provides a similar environment, but not an identical one with GitHub Workflows,
  • It works if you are creating workflows with existing and published versions of actions, but not if you develop one (I saw that lately they have worked to improve this),
  • Although it’s not impossible to spin-up in a CI environment, but I had the feeling that it’s somehow rather heavy-weight and meant just for local
  • Last, but not least, there are no assertions. You are able just to execute the workflow, but not more. It is meant to save you time by not committing small iterations over workflows, and save those precious GitHub Actions Minutes.

Introducing GitHub Actions Test Automation

After being sad that GitHub hasn’t provided the community with any out-of the box Testing Tools, I’ve decided to take this one on myself. This is how the GitHub Actions Test Automation or gat came to life. It is a very simple tool, a GitHub Action by itself, with the following features:

  • Runs inside the GitHub Workflows Ecosystem
  • Based on chai, can assert if an output should equal / should not equal to a specified value. 
  • That’s it, no other fancy stuff (apart from speaking some emoji :P).

Check it out: https://github.com/therussiankid92/gat

How to test your pipeline using gat

Let’s say we want to test this action.

Step 0: Learn about GitHub Actions

Actually, GitHub provides a pretty neat documentation.

Step 1: Understand the System in Test

This action checks if a google calendar is busy, and returns true or false accordingly. Given the following mandatory inputs: google-credentials, google-token, google-calendar-id, it outputs: calendar-busy 

Step 2: Design your Test Case

Let’s test the following:

When a calendar with no events is checked
Then the calendar is not busy

Step 3: Create your Test Data

Following the instructions in the README.MD file of the actions we are testing, create a calendar  with no events. For security, store all the inputs in GitHub secrets. 

Step 4: Decide running strategy for your test

I like to run them on push or on PR to master. Feel free to add the strategy you most fancy.

Step 5: Decide the assertion you want to use to run gat

In gat, you can find at the moment just two assertions: 

  • should.equal
  • should.not.equal

If you need something else, could you help the community with a PR?

Step 6: Create the workflow with the Test Automation

name: Happy Path - Returns false
on:
  push:
  pull_request:
    branches:
      - master
jobs:
  happy-path:
    runs-on: ubuntu-latest
    name: Returns false
    steps:
      - name: Given current branch is initialized
        uses: actions/checkout@v2
      - name: When a calendar with no events is checked
        uses: ./
        id: siesta
        with:
          google-credentials: ${{ secrets.google_credentials }}
          google-token: ${{ secrets.google_token }}
          google-calendar-id: ${{ secrets.google_calendar_id }}
      - name: Then the calendar is not busy
        uses: therussiankid92/gat@v1
        id: gat
        with:
          assertion: should.equal
          expected: false
          actual: ${{steps.siesta.outputs.calendar-busy}}

Step 7: Enjoy the magic

Conclusions: 

Using the above solution it is pretty straightforward, and can be applied to individual actions, complete workflows and to similar ecosystems. However, it opens an important topic around CI/CD: Testability. Most of the pipelines I have worked with (and even built myself until the very moment) are constructed to work just in production:

  • Poor to nonexistent analyzable outputs, they just do the heavy work and are focused on providing the final result (e.g.: push an artifact to Nexus), that can make it impossible to assert. If you are lucky, maybe you get some logs 🙂
  • They work just in close-to-real conditions (e.g.: even myself I needed to create a real calendar to execute the above test), that can make it impossible to execute frequently depending on what you are doing
  • Happy-paths, or insane debugging sessions on failure. Nothing in between.

As Pipelines have already become code, it makes me wonder if is it the right moment to take some actions together to improve the way we work with them?