XMPP and IoT

I plan to post alot more about the Internet of Things (IoT), but this post is just to get some thoughts down.

The more I read about the different competing protocols (XMPP, MQTT, DDS, AMQP, CoAP and others) the more I get concerned about the distinct lack of security (having been involved with the development of digital identity laws, it’s one of my pet rants).

Currently, the protocol receiving the most attention seems to be MQTT, but security is distinctly lacking. It’s basically a “vendor provided” feature (plain-text user name & password in an optional TLS encoded transport? This is HTTP basic-auth with all it’s issues). While this may be great for the large vendors who can stake out their ground and are large enough to fight off other contenders, it does nothing for interoperability (Single sign-on only works within a given vendor). Worse, the majority of smaller players will just skip the security, or do it badly, so that they can get an solution to market.

On the inverse side, I think that XMPP does provide a complete set of security options, but its more “complicated” set of standards (or XEP’s) cause users to turn away.

Additionally, XMPP gets a bad rap because the basic protocol is XML. <sarcasm> As everyone “knows”, XML is verbose. And isn’t this whole IoT thing about small constrained devices? </sarcasm> This line of reasoning always surprises me, since there is a never-ending list of failed predictions where some specific technological constraint caused people to say it’s impossible.

First of all, devices are becoming very powerful in a very small and cheap form factor. While there are still uses for very constrained devices, they, by their very nature, are more difficult to develop for than a more generic computer that can run more common languages and software. This problem constantly causes newer devices to be developed with more power and capability at the same cost as before. I’m currently playing with a Raspberry Pi, that at $35, provides a platform that is capable of sending tens of thousands of XMPP messages per second.

Secondly, there are many ways to reduce the verbosity of XML. Probably the best is the W3C EXI Recommendation, and it’s usage within XMPP (XEP-0322). Compressed XML via XMI can regularly achieve 100x or more reductions in size, thus reducing the verbosity argument to nothing.

But, in the end, this is just my opinion. The only way to help solve this problem is to actually contribute to the solution. Thus, I will be working on, talking about, and delivering IoT based solutions based on my take on the right solution, which, at the moment, is definitely based on XMPP.

Setting up a blog via Docker

My wife wanted a blog and I decided to start blogging as well as a way to document all the projects I work on.

Therefore, I figured that one of the first posts would be how I set up the blog system.

Of course, I could have just set up an account on the WordPress site, but where’s the fun in that? Instead, I wanted to build a reasonably complete Docker setup and have it deployed to a cloud provider.

Docker Installation

To start, I installed the latest (1.5 as of this post) build of Docker on one of my Linux environments. For the most part, I followed the instructions on:

https://docs.docker.com/installation/ubuntulinux/

I wanted to use the Docker-maintained package installation. My only real issue was that I run my own apt-cacher since I have dozens of virtual environments, and I found that the above instructions used an HTTPS update site that conflicted with apt-cacher. Instead of tracking down a better solution, I simply added

Acquire::HTTP::Proxy::get.docker.com "DIRECT";

to my apt configuration file at /etc/apt/apt.conf.d/01proxy

I also installed docker-machine from

https://docs.docker.com/machine/

and just copied the executable into /usr/local/bin

Setting up the Machine

I decided to try out using Digital Ocean as the cloud provider, since they had one of the cheaper setups ($5/month for a 512MB Memory / 20 Gig SSD / 1 TB data).

Once I had set up an account with Digital Ocean, I went to the Apps & API and then clicked “Generate new Token”.

DigitalOcean_Token

 

That generated a very long token key that I stored away. In the following code snippets, I’ll make it available as an environmental variable.

export DO_API_TOKEN=xxxxx

Next, I need to figure out which zone to start the machine in. Unfortunately, Digital Ocean’s website shows nice human-readable names, but not the codes needed by docker-machine. Fortunately, they do have a REST API that I could use.

curl -X GET -u "$DO_API_TOKEN:" https://api.digitalocean.com/v2/regions

This returned a JSON structure (formatted and truncated for readability):

{
 "regions" : [{
 "name" : "New York 1",
 "slug" : "nyc1",
 "sizes" : [],
 "features" : ["virtio", "backups"],
 "available" : false
 }, {
 "name" : "Amsterdam 1",
 "slug" : "ams1",
 "sizes" : [],
 ...

The ‘slug’ element is the one needed. So, if we wanted an box in New York, we’d use nyc1. In my case, I wanted San Francisco, so it was sfo1.

By running the docker-machine create command, we’re created a new machine in San Francisco, with only 512 MB of memory. The final parameter is the name for the machine, which I’ve called wordpress. NOTE: You start incurring charges at this point.

docker-machine create --driver digitalocean --digitalocean-access-token $DO_API_TOKEN --digitalocean-region sfo1 --digitalocean-size "512mb" wordpress

This takes a few minutes to provision and set up the machine,  and you should see something like:

INFO[0000] Creating SSH key...
INFO[0000] Creating Digital Ocean droplet...
INFO[0003] Waiting for SSH...
INFO[0066] Configuring Machine...
INFO[0108] "wordpress" has been created and is now the active machine.
INFO[0108] To point your Docker client at it, run this in your shell: $(docker-machine env wordpress)

Now, that you have a machine running, which you can verify at any time with:

docker-machine ls

NAME      ACTIVE DRIVER       STATE   URL                        SWARM
wordpress *      digitalocean Running tcp://104.236.140.57:2376

Or you can use the Digital Ocean dashboard to see that it’s running:

DigitalOcean_Running

 

Now, there are many different ways to issue the commands to the docker machine, but my favorite is to just set up the environment so that regular docker commands works with the remote machine. So, enter:

$(docker-machine env wordpress)

This sets a few environmental variables so that all future docker commands will run remotely.

Setting up the Reverse Proxy

One of the things that I wanted to do was run  many different websites from the same host. The easiest way to do that is to run a reverse proxy that internally forwards traffic based on the requested Host parameter to the appropriate docker container. I found a great docker container by Jason Wilder that did everything I needed:

https://registry.hub.docker.com/u/jwilder/nginx-proxy/

So, all I needed to do was to run his nginx reverse proxy container as the front-end.

docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock jwilder/nginx-proxy

Now, any new container that was registered with the right environmental variables would be automatically linked into the proxy container.

Setting up MySQL

Alot of the steps and inspiration for these instructions comes from a blog post:

http://antfie.com/super-easy-dockerised-wordpress/

Thanks out to Anthony Fielding for his set of instructions.

Because I want to be able to reproduce these steps for multiple different sites, I’ll store the site name and the MySQL password in environmental variables:

export SITE_NAME=diamondq
export MYSQL_PASSWORD=xxxx

I want to make it real easy to do management of the database environment, so I created a data container to hold the database data.

docker create --name $SITE_NAME-mysql-data -v /var/lib/mysql mysql

Next, I created the MySQL server itself.

NOTE: This is where I initially ran into a small challenge with the memory constrained environment at Digital Ocean. MySQL didn’t want to initialize because there wasn’t enough RAM. Since this was going to be a very low usage environment for some time, I just added 4 Gigs of disk space as swap. Connecting to the box was easy with the docker-machine command:

docker-machine ssh

Now, that I’m on the box, it was a few standard commands to generate the 4 gigabyte swapfile and make it permanent:

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile none swap sw 0 0" >> /etc/fstab

DOUBLE NOTE: When running the MySQL container that failed the first time, it left the data container in a bit of a mess, and I had to remove and restart the data container as well as the MySQL container to get it to work.

By providing some environmental variables, the wordpress database was automatically created.

docker run --name $SITE_NAME-mysql --volumes-from $SITE_NAME-mysql-data -e "MYSQL_ROOT_PASSWORD=$MYSQL_PASSWORD" -e "MYSQL_DATABASE=wordpress" -d mysql

We now have a MySQL database that is configured with an empty wordpress database. NOTE: I’m not exposing the MySQL port, so there is no way that anyone can remotely connect to the database.

Setting up WordPress

Now for WordPress. I originally looked around at the current WordPress container, but I wanted better control over the contents so that I could do things like back it up easier (as did Anthony, which is where I ran across his blog). So, I followed the same pattern as with MySQL. I started by creating a data container:

docker create --name $SITE_NAME-wordpress-data -v /var/www/html ubuntu

Next, I needed to download the current WordPress contents into this container. While this seems like a prime candidate for a separate Dockerfile, I didn’t want all this stored in the Docker AUFS, but instead in the Docker volume.

docker run -it --rm --volumes-from $SITE_NAME-wordpress-data ubuntu /bin/bash -c "apt-get install -y wget && cd /var/www/html && wget https://wordpress.org/latest.tar.gz && tar -xzf latest.tar.gz && mv wordpress/* . && rm -rf wordpress latest.tar.gz && chown -R www-data:www-data /var/www/html"

Since there may be multiple WordPress containers running on the same box, each ones needs a unique port number so that the nginx reverse proxy can talk to it. Thus, we’ll put the unique port number into another environmental variable:

export SITE_PORT=8080
export SITE_HOST=blog.diamondq.com

Now, we just start the final WordPress container:

docker run --name $SITE_NAME-wordpress --volumes-from $SITE_NAME-wordpress-data --link $SITE_NAME-mysql:mysql -p 127.0.0.1:$SITE_PORT:80 -e "VIRTUAL_HOST=$SITE_HOST" -e "VIRTUAL_PORT=$SITE_PORT" -d antfie/wordpress

This is where all the magic comes together. It brings in the WordPress data container so that the WordPress content itself is available. It links in the MySQL database, so it can be contacted internally via a virtual host called ‘mysql’. It exposes the Apache web server as port 8080, but only to the local host. This means that other containers can reach it (like nginx), but not external users. The two environment settings, VIRTUAL_HOST and VIRTUAL_PORT, are provided so that the nginx reverse proxy knows that it needs to link this container into the proxy. And finally, we’re using antfie/wordpress, since it’s basically the normal WordPress install with WordPress removed (since we’re bringing it in ourselves via the data container).

Assuming that your DNS is configured to point your website (blog.diamondq.com in my case) to the IP address of the Digital Ocean machine, everything should be ready to configure WordPress.

http://blog.diamondq.com

The only slight thing to know is that in one of the first configuration screens for WordPress, you must configure the database connection. In this case, the user name is root, the password is whatever is in $MYSQL_PASSWORD (although do not type $MYSQL_PASSWORD, since your browser is not aware of this environmental variable), and the host is the string ‘mysql’ (since that was the virtual host name when you linked in MySQL above).

Recap

For those who just want to do this quickly, it’s just three ‘blocks’:

Setup up the nginx reverse proxy

docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock jwilder/nginx-proxy

Define the environment you want to build

export SITE_NAME=diamondq
export MYSQL_PASSWORD=xxxx
export SITE_PORT=8080
export SITE_HOST=blog.diamondq.com

Build the environment

docker create --name $SITE_NAME-mysql-data -v /var/lib/mysql mysql

docker run --name $SITE_NAME-mysql --volumes-from $SITE_NAME-mysql-data -e "MYSQL_ROOT_PASSWORD=$MYSQL_PASSWORD" -e "MYSQL_DATABASE=wordpress" -d mysql

docker create --name $SITE_NAME-wordpress-data -v /var/www/html ubuntu

docker run -it --rm --volumes-from $SITE_NAME-wordpress-data ubuntu /bin/bash -c "apt-get install -y wget && cd /var/www/html && wget https://wordpress.org/latest.tar.gz && tar -xzf latest.tar.gz && mv wordpress/* . && rm -rf wordpress latest.tar.gz && chown -R www-data:www-data /var/www/html"

docker run --name $SITE_NAME-wordpress --volumes-from $SITE_NAME-wordpress-data --link $SITE_NAME-mysql:mysql -p 127.0.0.1:$SITE_PORT:80 -e "VIRTUAL_HOST=$SITE_HOST" -e "VIRTUAL_PORT=$SITE_PORT" -d antfie/wordpress

If you want a second site on the same box, then you just change the environment and re-run the build instructions. Takes about 30 seconds.

Building a new image on a Raspberry Pi

Even though there are many tutorials on how to initially set up a Raspberry Pi, I still found alot of them skipped steps or assumed too much knowledge. Since I tend to skitter between projects and come back to them months or years later, I figured I write this all down.

This is all based on a Raspberry Pi 2, but the majority of the steps should be similar for other versions; YMMV.

Raspberry Pi 2

Originally, my Pi had come with a 2 Gig microSD card, but I found that “way to small” to work with. Additionally, I wanted to have multi different projects on the go, so I wanted an easy to way to switch between the different projects. Therefore, I bought a 32 Gig SanDisk microSD from the local CostCo store for $30 CAD.

SanDisk 32Gb microSD

They’ve got a good video on the Raspberry Pi website for the basic install

http://www.raspberrypi.org/help/noobs-setup/

But I’ll jot down the key things here (additionally, I had to do a number of things to properly configure for a non-UK environment).

To format the SD card, download and install the SD Card Formatter software from the SD Association:

https://www.sdcard.org/downloads/formatter_4/

Put the card in the computer, and format the card. NOTE: A number of sites suggested that the FORMAT SIZE ADJUSTMENT be set to ON. Other’s didn’t mention it. I did change the adjustment, and it seems to work fine.

 

I downloaded the NOOBS image from the Raspberry Pi website:

http://www.raspberrypi.org/downloads/

and then simply unzipped the contents into the root of the freshly formatted SD Card.

After installing the card into the Raspberry Pi, and attaching all the cables to the keyboard, mouse, monitor and Ethernet, as well as power, it booted to the NOOBS OS selection page.

OSInstall

I don’t recommend choosing the Data Partition as well as the Raspbian image. Everytime you run the NOOBS installer again, it will wipe the Data Partition. There seems to be lots of people confused about the real ‘point’ of this option. As far as I can tell, it used if you have multiple OS’s installed at the same time, and want to have a shared partition between them.

I18N

Additionally, by default, the Raspberry Pi is configured for UK usage, which includes a UK keyboard. A quick shortcut to making it work properly is to change the Language during the install process (in my case to English (US) and a us keyboard.

The install of the OS takes about 10 minutes.

After a reboot, it drops into the raspi-config screen. I usually do a couple of changes to make the environment a little more tailored for me.

  1. Change User Password
    It’s amazing how many people leave the default password of raspberry. This creates a huge security problem. Please change the password to something more secure.
  2. Internationalization Options
    1. Change Locale
      I usually remove the en_GB locale, and add the en_US locale. Only having one locale selected speeds up the installation of many software, as well as reduces the amount of disk space consumed by each locale.NOTE: I always pick the UTF-8 variant, since not only does it enable good internationalization support, but also allows for much better terminal graphics support.Additionally, I set the only locale (en_US) as the default.
    2. Change Timezone
      I change the timezone to the appropriate one for me (America -> Vancouver)
  3. Finish

Now that raspi-config is finished, I usually force a reboot. Sometimes the I18N features are not fully working until a reboot.

sudo reboot

I usually do a full update cycle to make sure that all software packages are up to date with the latest.

sudo apt-get update

Followed by an upgrade to install any updated packages

sudo apt-get upgrade

At this point, the Raspberry Pi is ready to go.