Using GitHub actions and Vercel for end-to-end tests

End-to-end (E2E) tests are the tip of the test pyramid. They are supposedly the hardest to write and take the longest to run. But they are also valuable as they are the tests that "use" your app as your users do.

When we first started to use E2E tests at Signavio, we were able to run them after changes got merged to our master branch but not on feature branches. That is problematic because it had already made it to our mainline and possibly to production when they uncovered a defect. Another obstacle to not run them on each PR was that spinning up an environment was not an easy task and would take considerable time. Since each minute the pipeline runs prolongs the feedback loop for the individual developer, we had a problem.

When GitHub launched actions we discovered that actions can run when a deployment happens. By then, we already had set up Vercel to deploy our client on each PR. I asked myself whether I could connect the Vercel deployments with our E2E tests.

TL;DR<">

No, shortlist this time. Sorry! Suppose you are solely interested in connecting cypress to a Vercel deployment. In that case, you can skip ahead to the Creating the workflow section. If this is the first action you write, then I would recommend the full article. If you don't have time for that, definitively read Caveats. That might save you some time.

Vercel deployments<">

Setting up your project in Vercel is (and I don't say this lightly) easy if you're working with solutions like create react app or static site generators like Gatsby. Vercel can import and run your project without you needing to do any custom configuration. But even if you didn't bootstrap your page like that, the configuration is straightforward.

Another reason I'm pointing at Vercel is that they already integrate with GitHub and register themselves as a deployment which is essential for the next step.

GitHub actions<">

GitHub actions are a powerful tool. But I also find them a bit hard to get started with. I struggled a lot with wording and how to achieve a simple goal. This article is in no form a complete guide to writing good actions. It's merely a description of what I needed to do to get the job done.

Let's write an action! Wrong. The first thing I had to learn was that what I needed to do is add a workflow. I got fooled by the "Actions" tab on GitHub and assumed it would list actions. But what you will see there are workflows.

An action is one step in a workflow. A workflow composes actions so that you can achieve a more complex goal. You also need the workflow to express when you want these actions to run. Workflows are co-located with the code inside one repository. For GitHub to run a workflow, you need to create a YAML file inside the .github/workflows/ folder.

name: E2E tests
on: [deployment_status]

Why does this say deployment_status and not deployment?Because we wanted to improve the developer experience. To give our developers fast feedback on their PRs, we set a custom status. For that, we need some more information about the deployments.

Suppose you are not interested in building a custom action. In that case, you can skip ahead to Creating the workflow.

As we will need to work with the GitHub API to set a status on a PR, we will now create a custom action for that. GitHub offers two ways to define actions - JavaScript and Docker. In this example, we will create the action using Docker since this is how I know to do it. We will also use the actions-toolkit to make building custom actions easier.

An action to set a PR status<>>

Since we do not intend to release this action for other developers, we can put all code into a .github/actions/set-pr-status folder inside our repository. This folder will contain three files.

• a package.json because we want to use the actions-toolkit package,
• a DOCKERFILE that describes the container to run our code, and
• an action.js file that contains the code of our action

Let's start with the package.json. Since this will not be a package you want to release on NPM, we can keep it short.

{
  "name": "set-pr-status",
  "private": true,
  "main": "action.js",
  "dependencies": {
    "actions-toolkit": "5.0.0"
  }
}

Next, we need to set up the container for our action. If you have never worked with Docker before, this might look a bit confusing. An advantage of Docker is that other people have done the heavy lifting for us already and provided base images for containers (i.e., execution environments) that we can re-use. In our case, what we need to do is:

• use a base node image (because we want to execute JavaScript),
• copy all files we need into the container,
• install the dependencies, and
• run the action

FROM node:slim

# The * after package is there also to copy the
# package-lock.json that should is created when
# you run `npm install.`
COPY package*.json ./

RUN npm ci

COPY action.js /action.js

ENTRYPOINT ["node", "/action.js"]

We have defined the dependencies and made sure that an environment can run our action. Since we want to set a different PR status based on the workflow, the action needs two inputs.

1. The kind of status we want to set (pending, success, or failure)
2. A description that gives our developers some context

const { Toolkit } = require("actions-toolkit")

const tools = new Toolkit()

// You could also hard-code these as this
// action is bound to one repo but this makes
// copy-pasting this code easier
const { owner, repo } = tools.context.repo

// The SHA of the commit that triggered this action.
// Makes sure this status is associated with the
// correct commit.
const { sha } = tools.context

// These are inputs that we define. You can extend those
// or change their names if you like
const { state, description } = tools.inputs

tools.github.repos
  .createStatus({
    owner,
    repo,
    sha,
    state,
    description,
    target_url: `https://www.github.com/${owner}/${repo}/commit/${sha}/checks`,
  })
  .then(() => {
    tools.exit.success()
  })
  .catch((error) => {
    tools.log.fatal(error)
    tools.exit.failure()
  })

You might have noticed that we don't need to do any authentication with GitHub. The actions-toolkit library takes care of this for us as long as it finds a GITHUB_TOKEN environment variable with a valid token—more on this in the next section.

Creating the workflow<">

The primary goal of our workflow is to run our E2E tests. If you have skipped the section about adding a custom action, you can skip ahead to Pointing the test runner at the deployment

But it must also make sure that the correct states are shown in the PR. Here's what we want to achieve.

• Show a pending state while the deployment isn't ready or the tests are still running
• Show a success state when the tests have finished without errors
• Show a failure state when the tests failed

Let's start with setting the state to pending.

Registering a new status check<>>

We have defined our workflow to run whenever there is a deployment_status event. However, we solely want to set the PR status to pending as long as the deployment is also pending. Good thing jobs inside a workflow can be conditional.

jobs:
  set_pending:
    name: Register pending E2E tests state
    # This is the place where we define this job to only
    # run when the deployment state is still "pending".
    if: github.event.deployment_status.state == 'pending'
    runs-on: ubuntu-latest

    steps:
      # This checks out the code of this repository.
      # We need this because this is where our action
      # lives.
      - uses: actions/checkout@v1

      - name: Set status to "pending."
        uses: ./.github/actions/set-pr-status
        env:
          # You don't need to configure any secrets for this
          # to work. GitHub injects the GITHUB_TOKEN automatically.
          # We need it in this step so that our action
          # can talk to the GitHub API.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          # This is where we define the inputs
          # for this action.
          state: pending
          description: Waiting for E2E results

If we now open up a PR, we'll see a new status that reports on the status of our E2E tests.

Pointing the test runner at the deployment<>>

We're using cypress to run E2E tests. In this scenario, we're using the baseUrl configuration to point cypress to the location of the Vercel deployment. We also make sure that this job only runs when the deployment was successful.

jobs:
  run_e2e_tests:
    name: Run E2E tests
    # This statement makes sure that this job is only
    # executed when the deployment to Vercel was
    # successful
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    # Thank you cypress for providing a container
    # that works out-of-the-box
    container: cypress/browsers:node11.13.0-chrome73
    env:
      TERM: xterm

    steps:
      - uses: actions/checkout@v1

      - name: Run E2E tests
        run: cypress --config baseUrl=${{ github.event.deployment_status.target_url }}

Reporting the result of the tests back<>>

The last thing we need to do is to set the status to success when the tests passed and to failure when they did not pass. We can use the exit code of cypress for that. When a test does not succeed, cypress will exit with a non-zero exit code and mark the job as a failure.

jobs:
  run_e2e_tests:
  # See above for the complete definition of this job

  steps:
    - name: Set status to "success"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was successful
      if: success()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: success
        description: All tests passed

    - name: Set status to "failure"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was *not* successful
      if: failure()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: failure
        description: Some tests failed

That's it! We have created a custom action that we can use to set a status check on a PR and a workflow that will run our end-to-end tests when a deployment is ready.

Caveats<">

I like to mention one thing that probably cost me an hour right at the start. Because I was thinking in the context of a PR I always looked for the deployment action in the "Checks" tab of a single PR. But that is not how this works. Since deployments are not necessarily coupled to a PR, GitHub lists them in the "Actions" tab for the whole repository. I hope this piece of information saves you some time!