Distributed streaming applications are like Rube Goldberg machines. Lots of levers and knobs. You feel like you have to observe them in order to figure out how they work. This article is not about the one magic Spark parameter making your app scream. It's about how to approach optimizing your freshly developed Spark Streaming app.
A homogenous web development stack for heterogeneous environments
Developing and maintaining a mobile monetization platform like the Smaato RTB Ad Exchange, implied the use of Docker and Vagrant to set up a development stack, that could address some of our challenges. Projects like ours, with multiple developers spread across varying time zones can make setting up a local development environment a time consuming task. Add a team with different skillsets including a backend developer, a frontend developer and a designer working on the same codebase, then it gets tricky.