Quality Innovation In the Cloud at Boxever

While not the glitzy end of Boxevers’ business, I’d like to discuss a small piece of innovation, which enables us, a startup in Dublin, to punch above its weight and drive innovation that delivers for our customers.

Delivering Personalization at Scale

As the head of Quality at Boxever, I have the pleasure of leading a small team of amazing people that enable Boxever to deliver personalisation at scale to some of the biggest and most demanding organisations on earth. One of our customers was rated the World’s top Airline and another came 7th, according to Tripadvisor. We enable our customers to offer personalised services to over 1.2 billion customers.

The scale isn’t the only challenge, some of the design choice, which make the scale work, make traditional quality approaches too slow, too expensive and, well, too prone to failure. We’re multi-region so we’ve got peak traffic for some customer at every hour of the day with really no quiet time. Delivering an always active solution requires a large AWS investment. Our production clusters are expensive to maintain and testing on our production is not a good idea given our very active customer base.

Boxever’s Own Test Environment

Our solution, is to use our own test environment called Boxy. Boxy is an exact replica of our production environment with each microservice that makes up Boxever. It requires some funky networking configuration to live in AWS and allow multiple Boxy instances to auto deploy but it has allowed us to scale our testing. It also allows each engineer to test any issues with Boxever in minutes. We take snapshots during each test run so we can roll back any test run to an exact point to aid debugging.

Our production environment adheres to the highest security standards, which often makes debugging harder.  With Boxy, we can watch data as it passes thru the system.

In the first week of Boxy being available, our engineers started coming up with new use cases. One, which impressed me, was taking Boxy and simulating a rare but potentially nasty issue. He simulated a production issue on each Boxever micro service one by one until he saw the same problem.

This is something we could never do in production and the cost & time to replicate at production scale isn’t practical.

Boxy Customer Benefits

Our customers are seeing better quality, our developers are spending a lot less time on rework, which results in more and faster features at a lower cost. In the year I’ve been at Boxever, our APIs have always had 100% up time but, the difference is, it no longer requires the same late night heavy lifting it once did. We’re telling customers when they have problems. Our on-call engineers sleep a lot better. I cannot remember the last time I was paged.

The building blocks that we are using have been around for some time, containers were first released in 2005, I worked on ZFs in 2004 and, in 2017, we’re still innovating.