<iframe src="//www.googletagmanager.com/ns.html?id=GTM-MXN9JJ" height="0" width="0" style="display:none;visibility:hidden">

The Smaato Blog

Getting Started With User Acceptance Tests

Home » Blog » Getting Started With User Acceptance Tests
Posted by Gerd Rohleder on January 22, 2016
User_Acceptance_Tests.jpg

In our quest to develop the most efficient, full featured mobile-first advertising platform, it has become evident that Smaato's development team had to adopt user acceptance tests (UATs). Manual checks done directly in releases or integrations tests on small parts of the application wasn't sufficient enough to minimize risks for bugs. Besides, we couldn't afford to take a full day to do complete round trips or tests for a feature. UATs allowed us to run those crucial tests in an automated fashion and focus on quality.

How to Start

The fact that each application is “unique” to a developer prevents him from writing UATs for the frontend, backend and persistence. To avoid writing UATs, developers unconsciously tend to come up with excuses such as  “my application is too special”, “no one can just write a test to test everything”, “I have too many special rules”, “the setup is way too complex…”

Smaato’s development team’s obstacle was that we wanted to test everything right from the beginning, it was time consuming, and we didn’t have enough resources available. Since releasing new features is often more important than testing from a business point of view, the developer is expected to just implement the code without bugs. How difficult can this be?

Time to write UATs is often limited, therefore the trick is to start very small, one step at a time.

What Needs to Be Tested
User_Acceptance_Tests_at_Smaato.png

This is a simplified path through our application. When an incoming ad request from a client is valid, the settings of the Smaato Publisher Platform (SPX) select suitable DSPs for the publisher. Then all unsuited DSPs are filtered out, for example due to targeting. The remaining DSPs take part in the RTB auction. The winner from all bidding DSPs delivers the ad to the client. After the auction, we store the transaction data for internal workflows. This simplified path should be tested first as it's the most valuable, but other factors also need to be considered for more realistic paths:

  • The high number of DSPs taking part of the RTB auction
  • Other demand partner types like Ad Networks
  • Direct content or no ads available
  • Capping and publisher targeting
  • Network problems, high traffic, validation errors
  • and so on...

The First Test

The first test should request an ad from our test environment, check the response for success and verify the transaction data for the correct publisher, DSP and price. That’s all. We didn’t want to add test code into our application, everything should be done with our configuration.

A test publisher was created without any changes to SPX. This way all requests from this publisher would be processed with the default smaato line item. We then created a test DSP whose targeting only allows requests from the test publisher. This DSP takes part in all RTB auctions for the test publisher. All other DSPs were disabled.

At this point our biggest problem was on the DSP side since we had to mock the DSP’s responses. We chose to write the endpoint of the test DSP in Node.js.

Request

A publisher requests Smaato via a GET request and will receive a response as XML, HTML, JSON or JSONP. Htmlunit is the perfect tool to handle this.

Url url = RequestBuilder.build().withFormat(XML);
final WebRequest request = new WebRequest(url, HttpMethod.POST);
Page response = webClient.getPage(request
);
PageHandler pageHandler = new XmlPageHandler(response); 

The Builder creates a default URL where each parameter can easily be overwritten. In the example above, the created URL only differs from the default request URL in the request response format. All responses contain the same data, only the presentation differs. There’s no need to specify whether the response was XML or JSON since the PageHandler unifies the presentation. The test executes:

pageHandler.getSessionId()

DSP Response

Node.js is an open-source javascript runtime environment for server-side applications.It’s really easy to create a simple server which responds with predefined content.

This was our first Node.js application:

var http = require('http'),
   path = require('path'),
   fs = require('fs');;

http.createServer(function (req, res) {
   var fileName = path.basename(req.url);
   var content = fs.readFileSync('./response/' + fileName + '.xml', 'utf8');

   res.writeHead(200, {'Content-Type': 'application/xml'});
   res.end(content);
}).listen(4001, "127.0.0.1");

The files in the response directory contain the RTB responses from the DSPs as XML. Each DSP in a test has an endpoint like “https://test.smaato.net/rtbPrice2”, which will respond with the content of file rtbPrice2.xml

First Test Round

Everything for a simple first test is finished. We have a configuration which connects one DSP with the test publisher. The DSP replies with a predefined bid response and the test has the builder to create the necessary URL.

// setup
Url url = RequestBuilder.build().withFormat(XML);
final WebRequest request = new WebRequest(url, HttpMethod.POST);

// execution
Page response = webClient.getPage(request);

// verification
PageHandler pageHandler = new XmlPageHandler(response);
assertThat(pageHandler.getStatus(), is(SUCCCESS));

We were a bit surprised with how easy that was, and how fast we wanted to extend our UAT framework with the ability to validate the transaction data.

Transaction Data

The other important result of our application is the transaction data we use for internal workflows. This data is written in log files with a predefined structure. We use the sessionId as an identifier for a request.

We wrote a LogHelper to easily access this data. The helper retrieves the data for the sessionId from all relevant files, instantiating objects with this data and returns them to the test.

With this we can extend our verification to:

// verification

PageHandler pageHandler = new XmlPageHandler(response);

assertThat(pageHandler.getStatus(), is(SUCCCESS));

final LogRecord logRecord = getTransactionLogRecord(pageHandler.getSessionId());

assertThat(logRecord.getWinnerPrice(), is(WINNER_PRICE));

assertThat(logRecord.getWinnerDemandPartner(), is(WINNER)); 

Automatic Test Execution

We manually executed the first UATs. After we saw that we can trust the tests, we decided to automate their execution. We use Jenkins as continuous integration server. We built a flow to check out our project, compile it, deploy it to a test server and then execute the UATs. We use a plugin to create a copy of this flow for each feature branch. If a developer pushes code into a branch then a git hook executes the corresponding flow. With this we can be sure that no new code breaks a tested feature.

Conclusions

With this framework we have everything in place to write the most basic UATs. Since there was no time to write tests for all existing features and nobody wanted to do this, we’ve been writing tests only for new features. If we needed to extend our UAT framework then we would do it on the way.

It is important to resist the urge to implement everything that could be needed at once. Otherwise a lot of time would be invested in testing features which will not be used or at least not used how the developer thought about it.

Limitations

We had different problems with our UATs. Those were the most difficult:

  • A framework which supports a developer to write easy and fast UATs can become quite complex. Indeed, over time our framework code started to smell since whenever we needed something new, we just added as quick as possible. This bloat led to big problems understanding  what a test does and why a it failed. The clean design from the start quickly fell apart. We had to step back, rethink and redesign our framework a few times. Never forget to think carefully how you implement new features even if it’s only a framework to test!
  • UATs can create a false sense of security. If you have a UAT for every feature then no bugs can happen, right? Wrong! Of course they only tests what they test and nothing else. There were a few cases where bugs appeared in production because our UATs had not covered them. Developers can become fearless and careless with UATs which may result in more bugs. Be aware of that.

Key Learning

Whenever somebody wants to introduce UATs in a company and get responses like “no time for that”, “unnecessary” or “our application is too unique for automated tests”, find one basic test scenario which does not require a lot of work to test, but that is still helpful. Then create a test for this scenario as proof of concept. Perhaps a few tests need to be written before other developers start joining. But when they see how it goes and how much value they get, then they will do it. And it will contribute to good sleep if all your features are tested again and again and again…

 

Do you have any experience with UATs? How do you do it? Please write a comment and share your knowledge.

Written by Gerd Rohleder

Gerd is Smaato's Senior Java Developer in our platform's Core Technology team. He is a test-driven development practitioner and has extensive experience in behavior-driven development.

Want the latest in mobile advertising monetization strategies & ideas in your inbox? Subscribe today!