Using Artillery and Github actions for automated load testing

Introduction#

During one of our latest projects, our client told us they would be expecting a specific amount of requests in a short amount of time (think something like 5000 in 10 minutes). Even though the load is not excessive, we decided it would be best to be one hundred percent sure we would not be having trouble once those requests started pouring in. Then Martín and I started automating this task.

What we decided to do#

One of the important aspects of every application is its architecture. In our case, it being a web application, we were running a PERN stack (PostgreSQL, Express, ReactJS and Node.js) and hosting it in AWS, with S3 bucket + CloudFront for the frontend and ElasticBeanstalk for the backend.

One of the advantages of this solution is that we didn’t have to worry about our frontend availability, since S3 and CloudFront are responsible for it. So now we just need to check if our backend and database can handle the load. Enter: Artillery.io.

What Artillery does#

Artillery is a “modern, powerful & easy-to-use solution for load testing and functional testing”, so it can both check that the backend is able to handle the load and check that the response is what it’s expected to be. It works by running scenarios in phases, so you can choose to run a light phase with not so many requests per unit of time and increase the request rate to simulate user access peaks.

The scenarios are highly customizable as well, as you can not only define the target endpoint but also the probability that the scenario is run. Other options to customize the scenario is the payload of the request and the expected response (e.g.: the response status, content type and even expected response payload keys).

But what does an artillery test file look like?

Here we have a simple test that tries to GET two endpoints, one of them is behind a request limiter and the other one is not.

We define two phases, the first starts with four requests per second and in the next five seconds it ramps up to ten, the second one keeps on sending ten requests for three seconds.

In each phase there are two scenarios running, each one requesting a different resource. As you may have guessed, we don’t expect the /limited requests to work, since that endpoint is protected by a request limiter, so we define the expected response status code to be 429 (Too Many Requests). On the other hand, we expect /unlimited to always work, returning a 204 status code.

In our case, the flow we were trying to test was someone visiting a user’s profile and performing an action, so our test looked something like this:

So you see our flow has two requests instead of one, that’s because in our app, every action will be preceded by a user profile view, and it’s important that the server fulfills both requests sequentially.

Automating everything#

Initially, because we were close to the release date, we decided to manually run our load tests and build our results charts. This worked fine and we were able to ship the code with confidence. But then Martín and I decided to automate the process so we can make sure that the app will always be able to respond to a rapid succession of requests.

Thanks to GitHub actions, we can run tasks after some events take place in the repository. Some of these events are pushing a commit to a branch or opening a pull request, which are the events we are interested in.

Github Actions is a world of its own so we won’t go much in deep about how it works. If you want to know more about it, here's a good article you can start with.

So our GitHub workflow is as follows:

First of all, we pull the latest code and install the node dependencies with some of GitHub’s own actions. Then we use akhileshns’ action to deploy an app to Heroku. For this, we make use of GitHub secrets so that we can change our Heroku configuration without needing to deploy a new version of our workflow.

Last but certainly not least we run our tests. For this, we have our own script setup. Our script reads all the tests in the directory, then runs them one at a time and finally prints out a summary in list form. If any of the load tests fail then the script exits with a non-zero value and if all tests are successful then the exit value is zero. The end result is that the GitHub action will only pass if all tests pass, and you can see if a commit introduces breaking changes.

Other possible workflows#

We presented a load test approach to artillery and GitHub actions. Other things you can do is integrating the actions with your unit testing library (be it Jest, Mocha, or whatever library you decide to use). You can also use artillery to do unit testing along with your load tests.

We presented a load test approach to artillery and GitHub actions. Other things you can do is integrating the actions with your unit testing library (be it Jest, Mocha, or whatever library you decide to use). You can also use artillery to do unit testing along with your load tests.