Testing AWS Scala Microservices

Photo Credit: NatalieMaynor

Understanding The Problem

Our progression towards a sane testing setup began at our fall hackathon. I had a suspicion that our end to end tests were slowing us down but I wanted to lead with data. Thanks to the awesome work of Nate Tenczar we use a slack bot named Dibs to moderate access to a number of shared resources including our end to end test environment. Dibs is a pleasure to work with and on the side it dumps metrics into our internal “dogfood” system. (Yes, we use Localytics to monitor parts of Localytics.) After some quick digging, I found that it often took developers 3 hours to run the end to end tests. Since our build server clocked the tests at around 15 minutes per run this meant that folks were frequently rerunning the test suite multiple times due to flaky tests and broken infrastructure.

  1. The tests ran serially — we had only one shared test environment
  2. The tests were fragile — any team’s service could break the tests and block everyone else
  3. The tests were overused — the end to end test suite was a catch-all for anything with more scope than a unit test

A Solution Appears

With the problem more clearly understood we turned to the literature and found Martin Fowler’s excellent slide deck on Testing Strategies in a Microservice Architecture. His diagnosis was spot on:

AWS Dependencies

The question then turned to one of execution. The end to end tests were initially set up to address the difficulty of testing locally with all the necessary AWS service dependencies. While some might propose refactoring our systems to allow for injecting fake test dependencies we tend to take a more pragmatic approach to testing that doesn’t require abstracting away our underlying services just so we can test. This does blur the line between a unit and an integration test but we find it gives us higher code coverage with less time spent writing tests. 5 years ago we had no other options than to test directly against AWS services. Thankfully things have progressed since then.

Wrapping Up Loose Ends

With the AWS dependencies settled we turned towards dealing with SQL. Thankfully this is a bit more of a solved problem. We had been slowly migrating over to ScalikeJDBC as our SQL library of choice and their testing support is fantastic. We whipped up a small utility for loading schema and seed data and, after some small tweaks to our SQL queries, settled on H2 for our testing database.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ben Darfler

Ben Darfler

Engineering Leader, Father, Meditator, Elected Official, He/Him. Currently Director @ Honeycomb.io