Using GitHub actions and Vercel for end-to-end tests

Philipp Giese
July 21, 2020
9 min

End-to-end (E2E) tests are the tip of the test pyramid. They are supposedly the hardest to write and take the longest to run. But they are also valuable as they are the tests that "use" your app like your users do.

When we first started to use E2E tests at Signavio we were able to run them after changes got merged to our master branch but not on feature branches. That is problematic because when they uncovered a defect it had already made it to our mainline and possibly to production. Another obstacle to not run them on each PR was that spinning up an environment was not an easy task and would take considerable time. Since each minute the pipeline runs prolongs the feedback loop for the individual developer we had a problem.

When GitHub launched actions we discovered that actions can run when a deployment happens. By then we already had setup Vercel to deploy our client on each PR. This was when I asked myself whether I could the Vercel deployments with our E2E tests.

TL;DR

No short list this time. Sorry! If you are solely interested in how to connect cypress to a Vercel deployment you can skip ahead to the Creating the workflow section. Should this be the first action you write then I would recommend the full article. If you don't have time for that definitively read Caveats. That might save you some time.

Vercel deployments

Setting up your project to work with Vercel is (and I don't say this lightly) incredibly easy. If you're working with popular bootstrapping solutions like create react app or static site generators like Gatsby then Vercel can import and run your project without you needing to do any custom configuration. But even if you didn't bootstrap your page like that the configuration is straight forward.

Another reason I'm pointing at Vercel is that they already integrate with GitHub and register themselves as a deployment. This is important for the next step.

GitHub actions

GitHub actions are a powerful tool. But I also find them a bit hard to get started with. I struggled a lot with wording and how to achieve a simple goal. This article is in no form a complete guide to writing good actions. Its merely a description of what I needed to do to get the job done.

Let's write an action! Wrong. The first thing I had to learn was that what I needed to do is add a workflow. I got fooled by the "Actions" tab on GitHub and assumed it would list actions. But what you will see there are workflows.

An action is one step in a workflow. A workflow composes actions so that you can achieve a more complex goal. You also need the workflow to express when you want these actions to run. Workflows are co-located with the code inside one repository. In order for GitHub to run a workflow you need to create a yaml file inside the .github/workflows/ folder.

name: E2E tests
on: [deployment_status]

Why does this say deployment_status and not deployment? Because we wanted to improve the developer experience. To give our developers fast feedback on their PRs we set a custom status and for that we need some more information about the deployments.

If you are not interested in how to build a custom action you can skip ahead to Creating the workflow.

As we will need to work with the GitHub API to set a status on a PR we will now create a custom action for that. GitHub offers two ways to define actions - JavaScript and Docker. In this example we will create the action using Docker since this is how I know to do it. We will also use the actions-toolkit as it makes building custom actions easier.

An action to set a PR status

Since we do not intend to release this action for other developers we can put all code into a .github/actions/set-pr-status folder inside out repository. This folder will contain three files.

  • a package.json because we want to use the actions-toolkit package,
  • a DOCKERFILE that describes the container to run our code, and
  • an action.js file that contains the code of our action

Let's start with the package.json. Since this will not be a package you want to release on NPM we can keep it short.

{
  "name": "set-pr-status",
  "private": true,
  "main": "action.js",
  "dependencies": {
    "actions-toolkit": "5.0.0"
  }
}

Next, we need to setup the container for our action. If you have never worked with Docker before this might look a bit confusing. An advantage of Docker is that other people have done the heavy lifting for us already and provided base images for containers (i.e. execution environments) that we can re-use. In our case what we need to do is:

  • use a base node image (because we want to execute JavaScript),
  • copy all files we need into the container,
  • install the dependencies, and
  • run the action
FROM node:slim

# The * after package is there to also copy the
# package-lock.json that should is created when
# you run `npm install`
COPY package*.json ./

RUN npm ci

COPY action.js /action.js

ENTRYPOINT ["node", "/action.js"]

We have defined the dependencies and also made sure that there is an environment that can run our action. Since we want to set a different PR status based on what happens in the workflow the action needs two inputs.

  1. The kind of status we want to set (pending, success, or failure)
  2. A description that gives our developers some context
const { Toolkit } = require("actions-toolkit")

const tools = new Toolkit()

// You could also hard-code these as this
// action is bound to one repository but this makes
// copy-pasting this code easier
const { owner, repo } = tools.context.repo

// The SHA of the commit that triggered this action.
// Makes sure this status is associated with the
// correct commit.
const { sha } = tools.context

// These are inputs that we define. You can extend those
// or change their names if you like
const { state, description } = tools.inputs

tools.github.repos
  .createStatus({
    owner,
    repo,
    sha,
    state,
    description,
    target_url: `https://www.github.com/${owner}/${repo}/commit/${sha}/checks`,
  })
  .then(() => {
    tools.exit.success()
  })
  .catch((error) => {
    tools.log.fatal(error)
    tools.exit.failure()
  })

You might have noticed that we don't need to do any authentication with GitHub. The actions-toolkit library takes care of this for us as long as it finds a GITHUB_TOKEN environment variable with a valid token. More on this in the next section.

Creating the workflow

The primary goal of our workflow is to run our E2E tests. If you have skipped the section about adding a custom action you can skip ahead to Pointing the test runner at the deployment

But it must also make sure that the correct states are shown in the PR. Here's what we want to achieve.

  • Show a pending state while the deployment isn't ready or the tests are still running
  • Show a success state when the tests have finished without errors
  • Show a failure state when the tests failed

Let's start with setting the state to pending.

Registering a new status check

We have defined our workflow to run whenever there is deployment_status event. However, we solely want to set the PR status to pending as long as the deployment is also pending. Good thing jobs inside a workflow can be conditional.

jobs:
  set_pending:
    name: Register pending E2E tests state
    # This is the place where we define this job to only
    # run when the deployment state is still "pending".
    if: github.event.deployment_status.state == 'pending'
    runs-on: ubuntu-latest

    steps:
      # This checks out the code of this repository.
      # We need this because this is where our action
      # lives.
      - uses: actions/checkout@v1

      - name: Set status to "pending"
        uses: ./.github/actions/set-pr-status
        env:
          # You don't need to configure any secrets for this
          # to work. GitHub injects the GITHUB_TOKEN automatically.
          # We need it in this step so that our action
          # can talk to the GitHub API.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          # This is where we define the inputs
          # for this action.
          state: pending
          description: Waiting for E2E results

If we now open up a PR we'll see a new status that reports on the status of our E2E tests.

Pointing the test runner at the deployment

We're using cypress to run E2E tests. In this scenario we're using the baseUrl configuration to point cypress to the location of the Vercel deployment. We also make sure that this job only runs when the deployment was successful.

jobs:
  run_e2e_tests:
    name: Run E2E tests
    # This statement makes sure that this job is only
    # executed when the deployment to Vercel was
    # successful
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    # Thank you cypress for providing a container
    # that works out-of-the-box
    container: cypress/browsers:node11.13.0-chrome73
    env:
      TERM: xterm

    steps:
      - uses: actions/checkout@v1

      - name: Run E2E tests
        run: cypress --config baseUrl=${{ github.event.deployment_status.target_url }}

Reporting the result of the tests back

The last thing we need to do is to set the status to success when the tests passed and to failure when they did not pass. We can use the exit code of cypress for that. When a test does not succeed cypress will exit with a non zero exit code. This marks the job in the workflow as a failure.

jobs:
  run_e2e_tests:
  # See above for the complete definition of this job

  steps:
    - name: Set status to "success"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was successful
      if: success()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: success
        description: All tests passed

    - name: Set status to "failure"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was *not* successful
      if: failure()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: failure
        description: Some tests failed

That's it! We have created a custom action that we can use to set a status check on a PR and a workflow that will run our end-to-end tests when a deployment is ready.

Caveats

I like to mention one thing that probably cost me an hour right at the start. Because I was thinking in the context of a PR I always looked for the deployment action in the "Checks" tab of a single PR. But that is not how this works. Since deployments are not necessarily coupled to a PR, GitHub lists them in the "Actions" tab for the whole repository. I hope this piece of information saves you some time!