{"id":123,"date":"2020-07-08T08:00:00","date_gmt":"2020-07-08T06:00:00","guid":{"rendered":"https:\/\/danilov.es\/?p=123"},"modified":"2025-03-13T11:33:37","modified_gmt":"2025-03-13T10:33:37","slug":"testing-github-actions","status":"publish","type":"post","link":"https:\/\/danilov.es\/index.php\/2020\/07\/08\/testing-github-actions\/","title":{"rendered":"Testing GitHub Actions"},"content":{"rendered":"\n<p>A couple of months ago, I was publishing my first of a series of GitHub Actions and I was amazed: we finally have a way to stop reinventing the wheel in CI\/CD and finally use, build and maintain as a community small to big parts of our existing pipelines. As the GitHub Marketplace is growing each day, building workflows in a near future it will all be just like a simple puzzle: stick together already created actions, and voila!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Context<\/h2>\n\n\n\n<p>The complexity of most of the existing actions is pretty reduced, and the size of many of them is comparable to a larger script. For many people in our industry that are guided by craftsmanship this is not a reason not to Unit Test. However, not surprisingly, there are quite some actions out there with no tests whatsoever. From my perspective, this can bring along the following problematic, as in one of the most critical aspects of software development:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>We are relying on unreliable pieces of software<\/li><li>We are creating more code (the workflows) for which no automation coverage is provided<\/li><\/ul>\n\n\n\n<p>Indeed, the reliability can be improved by providing the Unit Tests in the first place. Problem solved. But what about:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Testing the action inside the actual GitHub Actions Ecosystem? Or inside any CI\/CD Ecosystem.<\/li><li>Testing an entire workflow?<\/li><li>Doing all of that with automation?<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">The Solution<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What was out there<\/h3>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"761\" src=\"https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-1024x761.jpg\" alt=\"\" class=\"wp-image-279\" srcset=\"https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-1024x761.jpg 1024w, https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-300x223.jpg 300w, https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-768x570.jpg 768w, https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-1536x1141.jpg 1536w, https:\/\/danilov.es\/wp-content\/uploads\/2020\/07\/luismi-sanchez-ZiuL3lWgQiM-unsplash-2048x1521.jpg 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>an accurate picture of what I found<\/figcaption><\/figure><\/div>\n\n\n\n<p>When I was looking, there was one tool that I could find on the internet, called <strong><a rel=\"noreferrer noopener\" href=\"http:\/\/github.com\/nektos\/act\" target=\"_blank\">act<\/a><\/strong>, that could help you with running GitHub Actions locally. The GitHub Marketplace was empty. Nevertheless, from what I have experienced at that time, it had the following downsides:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>It provides a similar environment, but not an identical one with GitHub Workflows,<\/li><li>It works if you are creating workflows with existing and published versions of actions, but not if you develop one (I saw that lately they have worked to improve this),<\/li><li>Although it\u2019s not impossible to spin-up in a CI environment, but I had the feeling that it\u2019s somehow rather heavy-weight and meant just for local<\/li><li>Last, but not least, there are no assertions. You are able just to execute the workflow, but not more. It is meant to save you time by not committing small iterations over workflows, and save those precious GitHub Actions Minutes. <\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Introducing GitHub Actions Test Automation<\/h3>\n\n\n\n<p>After being sad that GitHub hasn&#8217;t provided the community with any out-of the box Testing Tools, I&#8217;ve decided to take this one on myself. This is how the <strong><a href=\"http:\/\/github.com\/therussiankid92\/gat\">GitHub Actions Test Automation<\/a><\/strong> or <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/therussiankid92\/gat\" target=\"_blank\"><strong>gat<\/strong><\/a> came to life. It is a very simple tool, a GitHub Action by itself, with the following features:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Runs inside the GitHub Workflows Ecosystem<\/li><li>Based on chai, can assert if an output should equal \/ should not equal to a specified value.&nbsp;<\/li><li>That\u2019s it, no other fancy stuff (apart from speaking some emoji :P).<\/li><\/ul>\n\n\n\n<p><strong>Check it out<\/strong>: <a href=\"https:\/\/github.com\/therussiankid92\/gat\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/therussiankid92\/gat<\/a> <\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to test your pipeline using gat<\/h3>\n\n\n\n<p>Let\u2019s say we want to test <strong><a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/marketplace\/actions\/tf-siesta\" target=\"_blank\">this action<\/a>.<\/strong><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 0<\/strong>: Learn about GitHub Actions<\/h4>\n\n\n\n<p>Actually, GitHub provides a pretty neat <a href=\"https:\/\/docs.github.com\/en\/actions\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>documentation<\/strong><\/a>. <\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 1<\/strong>: Understand the System in Test<\/h4>\n\n\n\n<p>This action checks if a google calendar is busy, and returns true or false accordingly. Given the following mandatory <em>inputs<\/em>: google-credentials, google-token, google-calendar-id, it <em>outputs<\/em>: calendar-busy&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 2: <\/strong>Design your Test Case<\/h4>\n\n\n\n<p>Let\u2019s test the following:<\/p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<pre class=\"wp-block-code\"><code lang=\"markdown\" class=\"language-markdown line-numbers\">When a calendar with no events is checked\nThen the calendar is not busy<\/code><\/pre>\n<\/div><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 3<\/strong>: Create your Test Data<\/h4>\n\n\n\n<p>Following the instructions in the README.MD file of the actions we are testing, create a calendar&nbsp; with no events. For security, store all the inputs in GitHub secrets.&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 4<\/strong>: Decide running strategy for your test<\/h4>\n\n\n\n<p>I like to run them on push or on PR to master. Feel free to add the strategy you most fancy.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 5<\/strong>: Decide the assertion you want to use to run gat<\/h4>\n\n\n\n<p>In gat, you can find at the moment just two assertions:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>should.equal<\/li><li>should.not.equal<\/li><\/ul>\n\n\n\n<p>If you need something else, could you help the community with a PR?<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 6<\/strong>: Create the workflow with the Test Automation<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"yaml\" class=\"language-yaml line-numbers\">name: Happy Path - Returns false\non:\n  push:\n  pull_request:\n    branches:\n      - master\njobs:\n  happy-path:\n    runs-on: ubuntu-latest\n    name: Returns false\n    steps:\n      - name: Given current branch is initialized\n        uses: actions\/checkout@v2\n      - name: When a calendar with no events is checked\n        uses: .\/\n        id: siesta\n        with:\n          google-credentials: ${{ secrets.google_credentials }}\n          google-token: ${{ secrets.google_token }}\n          google-calendar-id: ${{ secrets.google_calendar_id }}\n      - name: Then the calendar is not busy\n        uses: therussiankid92\/gat@v1\n        id: gat\n        with:\n          assertion: should.equal\n          expected: false\n          actual: ${{steps.siesta.outputs.calendar-busy}}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 7<\/strong>: Enjoy the magic<\/h4>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/_9jjbT8q6D9Mi_gGhHskVE2gNWutkVPawTLsgAa0k3v9nJIwlbd4N0Bf-4oFWXvXBrR_9kbZtZYFC0M4PcsWGRtfqF9jDvVUSEA1xXfSY0DIe2NSC1vlMmD1Y_cC2zUZr6ETLqi1\" alt=\"\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusions:&nbsp;<\/h2>\n\n\n\n<p>Using the above solution it is pretty straightforward, and can be applied to individual actions, complete workflows and to similar ecosystems. However, it opens an important topic around CI\/CD: Testability. Most of the pipelines I have worked with (and even built myself until the very moment) are constructed to work just in production:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Poor to nonexistent analyzable outputs, they just do the heavy work and are focused on providing the final result (e.g.: push an artifact to Nexus), that can make it impossible to assert. If you are lucky, maybe you get some logs \ud83d\ude42  <\/li><li>They work just in close-to-real conditions (e.g.: even myself I needed to create a real calendar to execute the above test), that can make it impossible to execute frequently depending on what you are doing<\/li><li>Happy-paths, or insane debugging sessions on failure. Nothing in between.<\/li><\/ul>\n\n\n\n<p>As Pipelines have already become code, it makes me wonder if is it <strong>the right moment to take some actions together to improve the way we work with them?<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A couple of months ago, I was publishing my first of a series of GitHub Actions and I was amazed: we finally have a way&#8230;<\/p>\n<div class=\"more-link-wrapper\"><a class=\"more-link\" href=\"https:\/\/danilov.es\/index.php\/2020\/07\/08\/testing-github-actions\/\">Continue Reading<span class=\"screen-reader-text\">Testing GitHub Actions<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":277,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[40,4,41,42,45,39,43,3,46,44],"class_list":["post-123","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-thoughts","tag-actions","tag-automation","tag-ci-cd","tag-devops","tag-gat","tag-github","tag-pipelineascode","tag-test","tag-tools","tag-workflows","entry"],"_links":{"self":[{"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/posts\/123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/comments?post=123"}],"version-history":[{"count":17,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/posts\/123\/revisions"}],"predecessor-version":[{"id":280,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/posts\/123\/revisions\/280"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/media\/277"}],"wp:attachment":[{"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/media?parent=123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/categories?post=123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/danilov.es\/index.php\/wp-json\/wp\/v2\/tags?post=123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}