One year ago we in Smaato's data engineering team decided to use Scala programming language instead of Java for all new applications. This decision was driven by a number of reasons. First, the software we’ve been using in this space (Spark and Kafka) is predominantly developed in Scala. We need to have a good grasp on the language to work with these tools to contribute and extend them. Furthermore, our development team was pushing to use a modern language, and Scala's functional aspects lend themselves well to the kind of problems we have to solve.

Being able to seamlessly interface with Java code allows us to leverage our existing code base and work with other teams at Smaato. This is a good moment to reflect on the transition and talk about our learnings.

Distributed streaming applications are like Rube Goldberg machines. Lots of levers and knobs. You feel like you have to observe them in order to figure out how they work. This article is not about the one magic Spark parameter making your app scream. It's about how to approach optimizing your freshly developed Spark Streaming app.