A homogenous web development stack for heterogeneous environments
Developing and maintaining a mobile monetization platform like the Smaato RTB Ad Exchange, implied the use of Docker and Vagrant to set up a development stack, that could address some of our challenges. Projects like ours, with multiple developers spread across varying time zones can make setting up a local development environment a time consuming task. Add a team with different skillsets including a backend developer, a frontend developer and a designer working on the same codebase, then it gets tricky. For obvious reasons, a designer doesn’t want to set up a database and a webserver to see how his work affects the project. This is where Vagrant had a huge impact in making the setup of our development environment very easy. However, we still needed a simple way of doing deployments on a staging or even on a live server. Adding Docker helped solve these problems.
To understand how Docker and Vagrant worked in conjunction on the Smaato development environment for web projects, let’s start with why we needed this setup to begin with:
- We were using Django, MongoDB, and had some dependence on Node.js
- We needed a simple mechanism to set up our development environments and share them among developers who didn’t need knowledge of the underlying technology
- Our developers use Linux (Ubuntu), Mac OS X and Windows, so it was necessary to have a solution that works on all of these platforms
- We wanted the deployment phase to be as painless to our systems engineers as possible
- The goal was to have our development environment as close as possible to the staging and live environments
With this in mind, Vagrant seemed like the obvious choice to maintain the same base Linux stack across our environments. In addition, we needed to be able to create images once (or update them when needed), and share them across our development and deployment systems. Docker could fulfill that promise. Although it took time to achieve the image setup, we are happy with the results.
Your development environment needs may vary, but we hope sharing what worked for us will get you started in less than 2 hours.
We’ve set up a very simple example: the Django project (just a blank Django installation) with the required scripts and files to start Vagrant and Docker. The project is available at this repository. You are welcome to clone it and simply run the following commands in the root of the project:
$ git clone https://github.com/smaato/docker-quickstart.git
$ cd docker-quickstart
$ vagrant up
Please feel free to use the structure and scripts to build your own Vagrant/Docker setup.
What does the stack look like?
Our software stack is Django + MongoDB on the backend. For the frontend, its Angular.js built with Grunt. Usual culprits here, nothing fancy. We are using Ubuntu 14.04 on Vagrant. As we set up our project, shared folders with boot2docker did not function as seamlessly as we expected. Now boot2docker supports VirtualBox Guest Additions, but it did not make its way into our project yet.
For Vagrant to work, you need to have a Vagrantfile in the root of your project. It’s just a text file, which allocates a virtual machine (VM) using Virtualbox. For consistency reasons, we run VirtualBox even on Linux hosts. Inside this VM, we share (sync, in Vagrant terms) our project folder from our host OS. The Vagrantfile allocates a fixed IP address to our VM and starts off our own provisioner using a shell script. The docker provisioner is configured without any options so that Docker will be installed in the VM, but Vagrant will not start off any containers. The startup script starts each of our containers, and mounts the shared volumes from Virtualbox (which is in turn mounted from host). Thus we have a working development environment which runs with our code changes. We can SSH into a container, and set up our IDE to do the same for its interpreter (Python 2.7 installed by default).
The build process
Images backing the Docker containers are built using Docker’s configuration file, called Dockerfile. The build process involves using the
$ docker build command to execute a set of steps that will create an image. We use a helper script to execute our build functions that are in containers.sh. Our images are structured in a way to have a base image (Dockerfile) for the project which contains all of the dependencies. The rest of the images use this base container and run only one application per derived container.
In order to build the base image manually, you could follow these steps:
- Run a Docker container using the debian:jessie image:
$ docker run -it --rm debian:jessie
- Update, apt, upgrade as needed.
$ apt-get update && apt-get upgrade
- Install your project’s apt level dependencies
- Make a note of above commands in a text editor
- Install non-apt level dependencies, for example using pip, or gem, or npm.
- Make a note of these commands separately in the text editor
The Docker build process will cache
$ apt-get commands which is very handy when you are rebuilding images since it speeds up the process a lot. Once you have listed your installation/setup commands, all you have to do is create a Dockerfile and place these commands as Docker RUN instructions.
To create a Docker image out of the Dockerfile, save the Dockerfile into a folder and run the following command:
$ cd docker-base
$ sudo docker build -t <username>/base .
The username is simply your username on Docker registry. It may also be the URL to your private registry. We need the registry to push our built images so that other developers can pull the same images to run them. You want to build images as a separate process, then pull and run them on developers’ machines or on deployment servers. That is why our Vagrant boots with
$ startup.sh which only pulls images, rather than building them. The
$ build.sh is only used by the maintainer of Docker images to update and push them to a private Docker registry.
Once the base image is built, you can then build the backend (Dockerfile), ssh (Dockerfile) and webserver (Dockerfile) images. These images have an ENTRYPOINT instruction which tells Docker to execute a set of commands when the image runs. Depending on which Dockerfile you’re working with, that ENTRYPOINT instruction is simply the UWSGI, Nginx or SSH daemon. You will notice that the Dockerfiles for backend, SSH and webserver depend on the base image. Once these three images are built, you can run your development setup using
For the purpose of our example, we have already added a Vagrantfile and four Dockerfiles to the project. All you have to do now is to run
$ vagrant up in the application root after cloning it. The process is easy enough to follow through the scripts and understand.
For Smaato, working with Docker and Vagrant in conjunction proved to be a fast and easy solution to set up our development environment, but there is even more. A few days ago Docker announced new tools for orchestration and composition of docker-based environments. Unfortunately they are still in Alpha status, but the goal of building these environments is pretty famous at the moment. We hope these tips and learnings will help you building yours!
Co-author: Sumit Datta