All MongoDB releases, everywhere

One of my “official side projects” is the Go language driver for MongoDB, started a few years back while looking for a way to store data conveniently from the Go language while leaving aside some of the problems we have mapping code into table-based approaches.

Nowadays this is used in projects at Canonical, in the MongoDB tooling itself, and also in some of my own personal projects. For the personal server-side projects I’ve been using docker containers to conveniently deploy the database tooling, but this week when I went to update some of my older images pulled from the docker hub I found that the docker installed on that server was a bit too old, so it was time to update the servers.

But that got me wondering: what if I replaced all those containers by snaps? This would allow me to keep the convenience and safety of the isolation, while making a lot of things simpler. Unlike docker, snaps make the installed tooling more easily accessible to the host system (bins in the search path, processes as actual children from shell and systemd, etc), and can even use system resources directly assuming interfaces allow it (e.g. home files).

So I got into that, and perhaps ended up overdoing it a little bit. Based on my experience testing the driver, I really appreciate having all versions available for playing with, rather than just the latest one. This is how my local development system looks like right now:

$ snap list | grep 'Name\|mongo'
Name                  Version               Rev     Developer   Notes
mongo22               2.2.7                 1       niemeyer    -
mongo24               2.4.14                1       niemeyer    -
mongo26               2.6.12                1       niemeyer    -
mongo30               3.0.12                1       niemeyer    -
mongo32               3.2.7                 1       niemeyer    -
mongo33               3.3.9                 1       niemeyer    -

These are backed by upstream tarballs (snapcraft downloaded them pre-built from mongodb.com), and are all installed, running, and with tooling available for playing with. They are also published to the snap store which means you can easily make use of them locally as well. If you want to do that, here is a crash course on snaps and on how I packaged the database together specifically.

If you’re using Ubuntu, update to release 16.04 which has snaps working out of the box. For other distributions have a look at snapcraft.io to see what command must be run for ensuring it is available.

Then, pick your version and install it. For example, if you want to play with the features just announced at MongoDB World this week, you want the unstable version 3.3.9:

$ snap install mongo33
93.59 MB / 93.59 MB [============================] 100.00 % 1.33 MB/s 

Name     Version  Rev  Developer  Notes
mongo33  3.3.9    1    niemeyer   -

After that you already have the tooling available in your path (assuming you have /snap/bin there), and the daemon started. So go ahead and fire the client to talk to the database:

$ mongo33
MongoDB shell version: 3.3.9
connecting to: 127.0.0.1:3317/test
MongoDB server version: 3.3.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings: 
(...) 
(...) ** NOTE: This is a development version (3.3.9) of MongoDB.
(...) **       Not recommended for production.
> 

Note that the server started on the non-standard port 33017 to allow multiple versions to be running together. The pattern is that the snap mongoNN will run on port NN017.

The tools installed also follow a similar pattern. Where upstream uses mongo, mongod, mongodump, etc, the snaps will have them under /snap/bin as mongo33, mongo33.d, mongo33.dump, and so on.

The systemd unit, if you want to interact with it for restarting, improving, debugging, etc, is named snap.mongo33.mongod.service, as usual for any snap that contains a daemon (snap.<snap name>.<app name>.service).

The data for the process that runs under systemd lives in the the standard writable area that the confinement system opens up for snaps in general – in this specific case /var/snap/mongo33/common/. The common directory is reused untouched on updates across snap revisions, which implies snapd won’t copy the data over when doing such updates. This compromises slightly the update safety, but is worth it for bulk data that we don’t want to copy on every snap refresh.

So this ended up quite nicely. Next up is the Go server code itself, which will be packed as a snap too, for deploying it into the servers in a similar way. The exact details for how these snaps are being built are publicly available too.

If you want a hand doing something similar, we have some helpful people at #snappy on FreeNode and also [email protected]

@gniemeyer

Posted in Architecture, Design, Go, MongoDB, Project | Leave a comment

Leveling up snapd integration tests

Over the last several months there has been noticeable and growing pain associated with the evolving integration tests around snapd, and given the project goal of being a cross-distribution platform, we are very keen on solving this problem appropriately so that stability is guaranteed everywhere.

With that mindset a more focused effort was made over the last few weeks to produce a tool that can get the project out of those problems, and onto a runway of more pleasant stability. Despite the short amount of time, I’m very happy about the Spread project which resulted from this effort.

Continue reading

Posted in Architecture, Assembly, C/C++, Cloud, Design, Go, Hardware, Haskell, Java, Lua, Project, Python, Ruby, Snippet, Test | Leave a comment

Snappy Interfaces

As much anticipated, this week Ubuntu 16.04 LTS was released with integrated support for snaps on classic Ubuntu.

Snappy 2.0 is a modern software platform, that includes the ability to define rich interfaces between snaps that control their security and confinement, comprehensive observation and control of system changes, completion and undoing of partial system changes across restarts/reboots/crashes, macaroon-based authentication for local access and store access, preliminary development mode, a polished filesystem layout and CLI experience, modern sequencing of revisions, and so forth.

The previous post in this series described the reassuring details behind how snappy does system changes. This post will now cover Snappy interfaces, the mechanism that controls the confinement and integration of snaps with other snaps and with the system itself.

A snap interface gives one snap the ability to use resources provided by another snap, including the operating system snap (ubuntu-core is itself a snap!). That’s quite vague, and intentionally so. Software interacts with other software for many reasons and in diverse ways, and Snappy is a platform that has to mediate all of that according to user needs.

In practice, though, the mechanism is straightforward and pleasant to deal with. Without any snaps in the system, there are no interfaces available:

% sudo snap interfaces
error: no interfaces found

If we install the ubuntu-core snap alone (done implicitly when the first snap is installed), we can already see some interface slots being provided by it, but no plugs connected to them:

% sudo snap install ubuntu-core
75.88 MB / 75.88 MB [=====================] 100.00 % 355.56 KB/s 

% snap interfaces
Slot                 Plug
:firewall-control    -
:home                -
:locale-control      -
(...)
:opengl              -
:timeserver-control  -
:timezone-control    -
:unity7              -
:x11                 -

The syntax is <snap>:<slot> and <snap>:<plug>. The lack of a snap name is a shorthand notation for slots and plugs on the operating system snap.

Now let’s install an application:

% sudo snap install ubuntu-calculator-app
120.01 MB / 120.01 MB [=====================] 100.00 % 328.88 KB/s 

% snap interfaces
Slot                 Plug
:firewall-control    -
:home                -
:locale-control      -
(...)
:opengl              ubuntu-calculator-app
:timeserver-control  -
:timezone-control    -
:unity7              ubuntu-calculator-app
:x11                 -

At this point the application should work fine. But let’s instead see what happens if we take away one of these interfaces:

% sudo snap disconnect \
             ubuntu-calculator-app:unity7 ubuntu-core:unity7 

% /snap/bin/ubuntu-calculator-app.calculator
QXcbConnection: Could not connect to display :0

The application installed depends on unity7 to be able to display itself properly, which is itself based on X11. When we disconnected the interface that gave it permission to be accessing these resources, the application was unable to touch them.

The security minded will observe that X11 is not in fact a secure protocol. A number of system abuses are possible when we hand an application this permission. Other interfaces such as home would give the snap access to every non-hidden file in the user’s $HOME directory (those that do not start with a dot), which means a malicious application might steal personal information and send it over the network (assuming it also defines a network plug).

Some might be surprised that this is the case, but this is a misunderstanding about the role of snaps and Snappy as a software platform. When you install software from the Ubuntu archive, that’s a statement of trust in the Ubuntu and Debian developers. When you install Google’s Chrome or MongoDB binaries from their respective archives, that’s a statement of trust in those developers (these have root on your system!). Snappy is not eliminating the need for that trust, as once you give a piece of software access to your personal files, web camera, microphone, etc, you need to believe that it won’t be using those allowances maliciously.

The point of Snappy’s confinement in that picture is to enable a software ecosystem that can control exactly what is allowed and to whom in a clear and observable way, in addition to the same procedural care that we’ve all learned to appreciate in the Linux world, not instead of it. Preventing people from using all relevant resources in the system would simply force them to use that same software over less secure mechanisms instead of fixing the problem.

And what we have today is just the beginning. These interfaces will soon become much richer and more fine grained, including resource selection (e.g. which serial port?), and some of them will disappear completely in favor of more secure choices (Unity 8, for instance).

These are exciting times for Ubuntu and the software world.

@gniemeyer

Posted in Architecture, Article, Cloud, Design, Go, MongoDB, Project, Security | Leave a comment

Snappy Changes

As announced last Saturday, Snappy Ubuntu Core 2.0 has just been tagged and made its way into the archives of Ubuntu 16.04, which is due for the final release in the next days. So this is a nice time to start covering interesting aspects of what is being made available in this release.

A good choice for the first post in this series is talking about how snappy performs changes in the system, as that knowledge will be useful in observing and understanding what is going on in your snappy platform. Let’s start with the first operation you will likely do when first interacting with the snappy platform — install:

% sudo snap install ubuntu-calculator-app
120.01 MB / 120.01 MB [===============================] 100.00 % 1.45 MB/s

Continue reading

Posted in Architecture, Article, Cloud, Design, Go, Project, Security | Leave a comment

Snappy Ubuntu Core 2.0 is OUT

Ubuntu and Snappy community, it’s time to celebrate!

After another intense week and a long Saturday focused on observing and fine tuning the user experience, the development team is proud to announce that Snappy 2.0 has been tagged. As has been recently announced, this release of Snappy Ubuntu Core will be available inside Ubuntu proper, extending it with new capabilities in a seamless manner.

Continue reading

Posted in Architecture, Design, Go, Project | Leave a comment

mgo r2016.02.04

This is one of the most packed releases of the mgo driver for Go in recent times. There are new features, important fixes, and relevant
internal reestructuring to support the on-going server improvements.

As usual for the driver, compatibility is being preserved both with old applications and with old servers, so updating should be a smooth experience.

Release r2016.02.04 of mgo includes the following changes which were requested, proposed, and performed by a great community.

Enjoy!

Continue reading

Posted in Go, MongoDB, Project | Leave a comment

Announcing mgo r2015.10.05

Another stable release of the mgo Go driver for MongoDB hits the shelves, and this one brings some important improvements and fixes. As usual, it also remains fully compatible with prior releases in the v2 series.

Please read along for the complete list of changes.

Continue reading

Posted in Uncategorized | Leave a comment

mgo r2015.05.29

Another release of mgo hits the shelves, just in time for the upcoming MongoDB World event.

A number of of relevant improvements have landed since the last stable release:

Continue reading

Posted in Go, MongoDB, Project | Leave a comment

Readying mgo for MongoDB 3.0

MongoDB 3.0 (previously known as 2.8) is right around the block, and it’s time to release a few fixes and improvements on the mgo driver for Go to ensure it works fine on that new major server version. Compatibility is being preserved both with old applications and with old servers, so updating should be a smooth experience.

Release r2015.01.24 of mgo includes the following changes:

Continue reading

Posted in Go, MongoDB, Project, Uncategorized | Leave a comment

A timely coffee hack

It’s somewhat ironic that just as Ubuntu readies itself for the starting wave of smart connected devices, my latest hardware hack was in fact a disconnected one. In my defense, it’s quite important for these smart devices to preserve a convenient physical interface with the user, so this one was a personal lesson on that.

Continue reading

Posted in Assembly, Hardware, Project | 2 Comments