So, I’ve been doing some more reading on running a production docker environment, and it’s clear that docker really “stops” at the host level. Managing multiple hosts and multiple applications is a real hassle in the default docker environment.

Enter Weave.

It automatically provides a more robust network overlay that your docker containers can work with. I highly suggest that you read through their website.

I wanted to record the process I went through to set up a Weave environment across two different Cloud Providers (Digital Ocean and Vultr).

Vultr is not directly supported for the new docker-machine functionality, so I used their website to create a basic host using the Ubuntu 14.04 LTS version.

Once the host was provisioned, I connected using SSH, and setup and installed docker:

vultrbox% wget -qO- https://get.docker.com/ | sh

As of this article it was Docker 1.8, which is pretty close to the minimal version needed to get Weave working properly.

Back in my docker management box (at my house), I added the new box to management under docker-machine:

home% docker-machine create --driver generic --generic-ip-address vultrbox --generic-ssh-user root --generic-ssh-key .ssh/MikeMansell.priv vultr3

This box is now managed by docker-machine, which makes it much easier to issue commands.

The next step was to download the Weave commands onto the management box.

home% curl -L git.io/weave -o /usr/local/bin/weave
home% chmod a+x /usr/local/bin/weave

Since we’re going to issue all the next set of commands in the context of our vultr box, we’ll set up an environment variable so that it automatically connects there:

home% export DOCKER_CLIENT_ARGS="$(docker-machine config vultr3)"

Next, we need to set up the Weave agent, but we want it to be secure and to support multiple isolated applications. So, I ran:

home% weave launch -password $WEAVE_PASSWORD -iprange 10.2.0.0/16 -ipsubnet 10.2.1.0/24

This sets up a password that will be used to make sure that all traffic between the different Weave proxies are encrypted. It also says that Weave will manage the 10.2.0.0 subnet, and if an application is launched without explicitly defining a subnet to run it in the 10.2.1.0 subnet. This launches the proxy as a container on the vultr3 box.

Next, we want to have a DNS server running.

home% weave launch-dns

And finally, we want the Docker API proxy so that any docker container command are automatically routed through weave. However, since we want to make sure that everything remains secure, we’ll have to use the same TLS commands that were used to secure docker in the first place.

home% weave launch-proxy --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem

We now have three containers running on our vultr3 box. This represents our ‘base’ environment. We can do a quick test to make sure that everything is fine by redirecting our docker commands through weave.

home% eval $(weave proxy-env)

And now we can run a test command to see how it’s working.

home% docker run --name a1 -ti ubuntu
root@a1:/# hostname
a1.weave.local

We started a ubuntu shell container over on vultr3, but we can see that it’s run through weave, since it was assigned a DNS name in the weave.local DNS namespace.

We’ll clean that up.

root@a1:/# exit
exit
home% docker rm a1
a1

For a real scenario, we’re going to run a basic Cassandra server on this box. NOTE: We’ll run it in a separate isolated subnet.

home% docker run --name cassandra1 -e WEAVE_CIDR=net:10.2.2.0/24 -v /root/data/cassandra1:/var/lib/cassandra/data -d cassandra:2.2.0

Next, we want to get a Digital Ocean host set up, so that it can join the network. Similar to my other posts, I’ll assume that the Digital Ocean Acess Token is stored in an environmental variable.

home% export DO_API_TOKEN=xxxxx

Then, we’ll create the machine

home% docker-machine create --driver digitalocean --digitalocean-access-token $DO_API_TOKEN --digitalocean-image "ubuntu-14-04-x64" --digitalocean-region "nyc3" --digitalocean-size "512mb" donyc3

This creates a new Docker host running in New York, with the cheapest hosting plan.

We’ll switch over to using this host for the next set of commands

home% eval $(docker-machine config donyc3)
home% export DOCKER_CLIENT_ARGS="$(docker-machine config donyc3)"

And, we’ll launch the Weave environment here as well. The big change is that when we launch Weave, we’ll provide the host of the vultr3 box so that they can connect together.

home% weave launch -password $WEAVE_PASSWORD -iprange 10.2.0.0/16 -ipsubnet 10.2.1.0/24 $(docker-machine ip vultr3)
home% weave launch-dns
home% weave launch-proxy --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem

Now, we can use the Docker proxy for this second box:

home% eval $(weave proxy-env)

And lastly, we can run a Cassandra query from Digital Ocean in NYC to the Cassandra server box over in Vultr.

home% docker run --name cq -e WEAVE_CIDR=net:10.2.2.0/24 -ti cassandra:2.2.0 cqlsh cassandra1

From the point-of-view of the containers, they think they are running in the same network. That’s pretty cool.