Smaato operates one of the world’s largest mobile advertising marketplaces. Our platform handles up to 10 billion mobile ads every day around the world. One of the challenges we, as Smaato developers, face daily is successfully integrating and communicating with the vast variety of technical interfaces used by our many partners who transact on our marketplace platform.
In this blog post, we wanted to give the technically inclined among you a glimpse into the challenges and successes we face in solving issues, in this case, related to monolithic applications by implementing extensible components using the Groovy language and also discuss why our developers chose Groovy over other available options.
Smaato, as a leading mobile advertising platform for mobile publishers and app developers, engages with over 10,000 advertisers through ad networks and DSPs.
Up to 115,740 times per second, the Smaato Publisher Platform (SPX):
- receives an ad request from a smartphone anywhere in the world (as a single mobile user interacts with a mobile website or an app on their mobile phone),
- asks interested advertisers (the demand side of the marketplace) to advertise to this specific user,
- runs an auction for the particular ad space among interested advertisers, and,
- (in less than 200 milliseconds), responds to the user’s smartphone with the winning ad.
In illustration terms, a publisher’s ad request through the Smaato advertising platform might look like this:
On both sides of our marketplace - the supply side (app developers and mobile publishers) and the demand side (advertisers, media agencies and trading desks through DSPs and ad networks), the Smaato platform uses HTTP for communication. On the supply side, the Smaato platform (Smaato Open Mobile Advertisement Platform or SOMA Platform) only exposes our API (SOMA API on Smaato wiki). In contrast, on the demand side, SOMA has to implement a variety of interfaces provided by demand partners.
Very little effort is needed to integrate RTB Demand Partners whose systems must implement the OpenRTB specification. However, more effort is required to implement ad networks’ interfaces that vary from one ad network demand partner to another. Consider the following:
AdNetwork A accepts ad requests over HTTP GET and responds with JSON/XML/HTML AdNetwork B accepts ad requests over HTTP POST with JSON body and responds with JSON AdNetwork C accepts ad requests over HTTP POST with XML body and responds with HTML AdNetwork D accepts ad requests over HTTP POST with PROTO data and responds with PROTO
And so on…
Obviously, every JSON/XML/Proto schema is unique and SOMA should support them all. In addition, demand partners often change/ improve/ break/ enrich their APIs.
Let’s take a closer look at a Connectors component - an incoming ad request from an ad network:
We can clearly identify 3 operations that define the internal contract:
- Request mapping
- Response mapping
Sending requests and receiving responses is taken care of by the HTTP client.
Obviously, every JSON/XML/Proto schema is unique, and SOMA aims to support them all. In addition, demand partners often change/improve/break/enrich their APIs.
As we worked with all of these many interfaces, we faced (and sought to solve) the following problems:
- The SOMA team lacked a reliable, unerring approach of implementing or changing demand partner connectors. We needed a (micro) framework that would allow us to satisfy any possible API implementation change/request.
- The development team experienced incidents where they had a vast number of connectors implemented with many of them having been written years ago. We realized that maintaining these connectors is inefficient and that a slight change inside a legacy connector could break a whole connector.
- Turnaround times were slow.
- The process of implementing a feature request for an ad network connector implementation/change/fix could easily last from a couple of weeks to a couple of months. Due to the SOMA specific deployment process, which includes A/B testing, it might take up to two weeks to release the implemented connector. We needed faster turnaround times and wanted connectors to take effect instantly.
- Processes and implementation were cumbersome.
- We wanted Sales Engineers to easily be able to apply changes into the framework with minimal programming necessary, modify connectors on their own and implement APIs simultaneously with feasibility checks.
- The team didn’t want to throw away working, already existing connectors.
- The framework had to provide the possibility to extend, fix or change existing connectors.
The team evaluated the following technological approaches:
- Object graph query/manipulation framework, where a publisher request object was mapped to an ad network request object, and the ad network response was mapped to a publisher response:
- In-house written mapping definition created at compile time.
- SpringEL, with its own runtime expression compiler.
- Microservices way: Every connector ran as a separate microservice, at the same time implementing an API to communicate with SOMA.
- Java classloader: With this approach, we stored our connector classes somewhere outside and attached them to the system during its runtime. (Plugin architecture)
- Groovy classloader: Pretty much like Java classloader, except that it's with Java-on-steroids.
All of the above have certain pros and cons:
- The custom implementation of object graph frameworks that we came up with was not that flexible and extendable. However, it was possible to implement fast.
- Although SpringEL was promising, micro-benchmarking showed that it was too slow for our needs (might add an overhead of couple of milliseconds).
- Microservices added some latency overhead, and the price for maintaining all the infrastructure was too high (Smaato has 400+ Ad Networks, and that number is continually growing).
- Java and Groovy classloaders looked similar, but metaprogramming, room for DSL, and libraries for building and parsing JSON/XML/HTML made groovy a clear winner.
The so-called Connector Definitions framework is constructed out of the following components:
- Groovy class loader: Loads plugin and puts it into a cache.
- Caching mechanism: Reloads classes only when source code is changed.
- Compilation Customizers (especially SecureASTCutomizer): Prevents accidental usage of potentially harmful code (java.io.* , sun.unsafe.* , java.lang.System etc.).
- Existing Ad Network Connector module: Accepts legacy connectors and newly written pluggables
- Helper classes exported from module to plug-ins, so that common code is reused.
- Imported NekoHTML into module for HTML parsing within connector plugins
- Testing mechanism to prevent harmful/buggy plugins in production
- Source code delivery mechanism. We wanted to tag and version our plugins as they are changed, so that the recovery of broken plugins is fast and painless.
The overall framework design could be described as illustrated below:
Don’t be misled by the name of SecureASTCustomizer. Even if there is ‘Secure’ in its name, it cannot prevent users from executing malicious code. SecureASTCustomizer works statically with syntax tree, and there are plenty of ways to trick it. Still, if you can sacrifice a bit of performance, consider using Dynamic Type Checking Extensions on top of SecureASTCustomizer. For additional perspective, Kohsuke Kawaguchi wrote a nice blogpost in 2014 about why SecureASTCustomizer isn’t about security.
We were aware that class unloading is problematic and could be a pain point for us. After running numerous tests, we discovered that uncontrolled compilations of Groovy scripts could end up having ~1M classes piled in metaspace within a couple of days, which would, of course, spawn some performance issues.
To solve the problem described above, we designed the following approach:
- Plugins in SOMA are stored in a custom registry, where we store connectors as classes compiled via GroovyShell using application classloader.
- Registry is designed not to compile Groovy scripts in case there were already compiled and stored in the registry. Map.computeIfAbsent(<Key>, <Function>) basically did the trick for us. Here Key stands for groovy code as String, and Function is GroovyShell.parse(<Key>).
This project was successful in solving the problem it was meant to address:
- We simplified how changes to ad networks' API implementations are made.
- We created, implemented and released a framework which supports ~99.5% of connector APIs. What about the remaining 0.5%, you ask? Those are rare and exceptional cases when we have to implement an e.g protobuf-based connector or something even more exotic.
- Whenever new connectors are deployed, it now takes at most 10 minutes for changes to take effect.
- Most of changes inside a connector definition framework are now handled by our Sales Engineers, who appreciate that they can now troubleshoot demand partners’ requests themselves. Still, we are eager to go one step further by introducing a sort of DSL and improve work with connectors even more.
- Best of all, we accomplished all of this without changing a single line of code inside existing connectors.
It is also worth mentioning that our OpenRTB implementation was also rewritten in Groovy, where JsonBuilder made implementation of RTB extensions cleaner. Nonetheless, JsonBuilder itself wasn’t designated for extensions or modification, and we hope will be addressed in future releases of Groovy.
We’d like to hear from you about your own experiences and invite you to share them here.