Mocker for Python released!

Posted on 2007/11/11 by niemeyer

After being bored for a long time for the lack of a better infrastructure for creating test doubles in Python, I decided to give it a go.

I’m actually quite happy with what came out.. it took me about four weekends (was developed as a personal project), and I’ll dare to say that it’s the best mocking system for Python at the present time. Not only that, but it has features that I’ve not seen in any other mocking/stubing infrastructure, independent of language.

Here’s a feature list to catch your attention:

  • Graceful platform for test doubles in Python (mocks, stubs, fakes, and dummies).
  • Inspiration from real needs, and also from pmock, jmock, pymock, easymock, etc.
  • Expectation of expressions defined by actually using mock objects.
  • Expressions may be replayed in any order by default,
  • Trivial specification of ordering between expressions when wanted.
  • Nice parameter matching for defining expectations on method calls.
  • Good error messages when expectations are broken.
  • Mocking of many kinds of expressions (getting/setting/deleting attributes, calling, iteration, containment, etc)
  • Graceful handling of nested expressions (e.g. ”person.details.get_phone().get_prefix()”)
  • Mock ”proxies”, which allow passing through to the real object on specified expressions (e.g. useful with ”os.path.isfile()”).
  • Mocking via temporary ”patching” of existent classes and instances.
  • Trivial mocking of any external module (e.g. ”time.time()”) via ”proxy replacement”.
  • Mock objects may have method calls checked for conformance with real class/instance to prevent API divergence.
  • Type simulation for using mocks while still performing certain type-checking operations.
  • Nice (optional) integration with ”unittest.TestCase”, including additional assertions (e.g. ”assertIs”, ”assertIn”, etc).
  • More …

Worked? Check it out!

Posted in Project, Python, Test | Leave a comment

Python’s os.environ

Posted on 2007/10/17 by niemeyer

As Chris Armstrong pointed out yesterday, os.environ.pop() is broken in Python versions at least up to 2.5. The method will simply remove the entry from the in-memory dictionary which holds a copy of the environment:

>>> import os
>>> os.system("echo $ASD")
 
0
>>> os.environ["ASD"] = "asd"
>>> os.system("echo $ASD")
asd
0
>>> os.environ.pop("ASD")
'asd'
>>> os.system("echo $ASD")
asd
0

I can understand that the interface of dictionaries has evolved since os.environ was originally planned, and the os.environ.pop method was overlooked for a while. What surprises me a bit, though, is why it was originally designed the way it is. First, the interface will completely ignore new methods added to the dictionary interface, and they will apparently work. Then, why use a copy of the environment in the first place? This will mean that any changes to the real environment are not seen.

This sounds like something somewhat simple to do right. Here is a working hack using ctypes to show an example of the behavior I’d expect out of os.environ (Python 2.5 on Ubuntu Linux):

from ctypes import cdll, c_char_p, POINTER
from UserDict import DictMixin
import os


c_char_pp = POINTER(c_char_p)


class Environ(DictMixin):

    def __init__(self):
        self._process = cdll.LoadLibrary(None)
        self._getenv = self._process.getenv
        self._getenv.restype = c_char_p
        self._getenv.argtypes = [c_char_p]

    def keys(self):
        result = []
        environ = c_char_pp.in_dll(self._process, "environ")
        i = 0
        while environ[i]:
            result.append(environ[i].split("=", 1)[0])
            i += 1
        return result

    def __getitem__(self, key):
        value = self._getenv(key)
        if value is None:
            raise KeyError(key)
        return value

    def __setitem__(self, key, value):
        os.putenv(key, value)

    def __delitem__(self, key):
        os.unsetenv(key)

I may be missing some implementation detail which would explain the original design. If not, I suggest we just change the implementation to something equivalent (without ctypes).

Posted in Python, Snippet | 2 Comments

Landscape and Storm go public

Posted on 2007/08/15 by niemeyer

Finally, a couple of projects I’ve been working on in the last year and a half have been made public, which means that I have more freedom to talk about them openly.

Landscape

Landscape is a system we’ve created to allow administrators to comfortably manage and observe a large number of computers remotely through a centralized web interface.

This description certainly won’t strike anyone as a brand new idea. There are indeed a large number of systems for remote management. Even then, Landscape does bring new ideas into that known field, such as a very flexible package management offering. Landscape, supporting only Ubuntu at the present moment, also has the advantage of being built inside the company which supports the operating system distribution itself.

There are currently 5 core developers, with many other people contributing in various areas. My role is being a Technical Lead, even though that says very little about the kind of relationship that we have within the project. The guys I work with are very smart and goal oriented, so decisions are taken through friendly discussions and consensus, and initiative is seen coming from all directions.

Storm

Storm is a ORM we have developed for Python, to be used in Landscape, Launchpad, and other projects. The project was originally started because our attempts to perform client side partitioning (sharding) of data with existent ORMs for Python failed.

It was announced as an open source project in a talk I presented last month at EuroPython, and last week the second public release (0.10) was already made.

If you are around the Boston area in the US, my coworker and friend Christopher Armstrong will be giving a Storm talk at the Cambridge Python Meetup today. I’ll also be presenting it again at PyCon Brasil at the end of the month, in Joinville, Brazil.

Posted in Conference, Project, Python | 3 Comments

python-dateutil 1.2 released

Posted on 2007/06/26 by niemeyer

python-dateutil version 1.2 has just been released.

It includes the following changes:

  • Now tzfile will round timezones to full-minutes if necessary, since Python’s datetime doesn’t support sub-minute offsets (reported by Ilpo Nyyssönen).
  • Removed bare string exceptions (reported and fixed by Wilfredo Sánchez Vega)
  • Fixed bug in leap count parsing (reported and fixed by Eugene Oden).
Posted in Project, Python | 2 Comments

Smart 0.51 available

Posted on 2007/05/20 by niemeyer

Smart 0.51 has been released today. It includes a few bug fixes and some minor updates.

Shortly after the release, I’ve added a couple of new hooks on Smart’s trunk as well: cache-loaded, and cache-loaded-pre-link. These should enable people to write plugins that hack the cache for specific purposes. Axel Thimm has requested these for a while to introduce kernel-related upgrades. Hopefully these will fulfill his needs.

This release took a while.. probably because I’ve been quite immersed in our current project at Canonical, traveling very frequently, and without much time to blog or to do some of the usual open source activities I used to. The good thing is that we’re getting very close to the public announcement, and some of the work we’ve been doing will be released as open source, so we’re all likely to get more community-oriented interactions again.

Posted in Project, Python | Leave a comment

Quickies

Posted on 2007/03/10 by niemeyer

brother…

My brother Diogo is in town! Good to see him after so much time.

pycon…

PyCon 2007 was fantastic. It was great to meet everyone there, and we had two awesome sprinting weeks around it.

confluence…

I’ve recently visited a confluence with a good friend of mine. Kayaks, paddling, walking, driving, swimming, aslphalt, sand, water, grass.. it was awesome.

svn2bzr…

It looks like Bazaar tags are now really coming, so I’m doing some work on svn2bzr again. Hopefully this time I’ll really migrate some projects over.

editmoin…

Version 1.9 of editmoin was released.

smart…

Some work in Smart is coming in the upcoming weeks.

projects…

Hopefully I’ll be able to speak more openly about (some of the) interesting things I’ve been working on in the near future.

Posted in Conference, GPS, Other, Project, Python, RCS | Leave a comment

editmoin now talks to moin 1.6

Posted on 2006/11/20 by niemeyer

editmoin 1.8 was just released, including support for moin 1.6, submitted by both David I. Lehn and Daniele Favara, and URL aliases, as suggested by Diogo Matsubara.

One interesting thing I should mention is that Daniele has sent me a reference to a Bazaar branch containing the change. It’s the first time I receive a bzr (as it’s also known) branch when the upstream project is using a different RCS (Subversion).

I actually intend to port this and other projects to Bazaar, but I’m waiting for the tags support. Hopefully it’s not too far away now.

Posted in Project, Python, RCS | 6 Comments

All Hands is over

Posted on 2006/11/20 by niemeyer

The All Hands conference from Canonical is over. It was one amazing week in San Francisco, US. Seeing everyone was really great. A lot of new people, many new ideas, old friends… It’s amazing how we get pumped in these conferences.. ;-)

Posted in Conference | Leave a comment

Smart upgrading algorithm

Posted on 2006/11/08 by niemeyer

One of the known issues in I’ve been trying to address in Smart for a while is the freezing effect that happens when a very complex upgrading situation (such as a full distribution upgrade) results in a combinatory explosion due to the number of choices to be analysed. Unfortunately, I never had time to really put in practice a reasonable solution for the problem. At this point, the beauty of open source software starts to shine.

A few weeks ago, Eran Tromer got close to the project and started researching and discussing about the issue. Not only that, but he produced actual patches that change the algorithm to prune the search space and find resonable solutions in acceptable timings. These patches were applied into the development version, and included in release 0.50rc1.

The preliminary results are quite impressive. David Farning has tested Smart 0.50rc1 with Fedora in several situations, and reported:

fc4->fc5, fc4->fc6, fc4->devel, fc5->fc6, all calculate updates and upgrades in a few minutes on a vm with 512M, using standard repos + Freshrpms and Dries. Much quicker than with 0.42.

This is really awesome. Thanks Eran!

Posted in Project, Python | 5 Comments

Released editmoin 1.5 and patcher 0.6

Posted on 2006/08/07 by niemeyer

New versions of editmoin and patcher were just released. They fix a couple of issues found by Jan Anlauff and Olivier Thauvin, respectively.

Posted in Project, Python | 1 Comment