Archive for the 'Python' Category

Watch out for list(dict.keys()) in Python 3

Published
by
Gustavo Niemeyer
on 2008-06-27
in Python
. 10 Comments

As everyone is probably aware by now, in Python 3 dict.keys(), dict.values() and dict.items() will all return iterable views instead of lists. The standard way being suggested to overcome the difference, when the original behavior was actually intended, is to simply use list(dict.keys()). This should be usually fine, but not in all cases.

One of the reasons why someone might actually opt to perform a more expensive copying operation is because, with the pre-3.0 semantics, the keys() method is atomic, in the sense that the whole operation of converting all dictionary keys to a list is done while the global interpreter lock is held. Thus, it’s thread-safe to run dict.keys() with Python 2.X.

The suggested replacement in Python 3, list(dict.keys()), is not. There’s a chance that the interpreter will give another thread a chance to run before or during the iteration of the view, and this will cause an exception if the dictionary is modified at the same time. To fix the problem, either a lock must protect the iteration, or a more expensive operation such as dict.copy().keys() must be used.

The 2to3 tool won’t help you there, unfortunately. So, keep an eye on it!

Google using Geohash

Published
by
Gustavo Niemeyer
on 2008-05-20
in Project and Python
. 6 Comments

According to Dave Troy, Google seems to be using the Geohash algorithm:

Google is employing the GeoHash algorithm I’ve been pushing to do spatial searching using BigTable. Since database schemes like BigTable don’t support traditional GIS extensions/spatial indexes, GeoHash allows for a simple bounding box search using truncated GeoHash substrings. I will post separately about this shortly, as I am working on some GeoHash tools to expand this functionality. This is of particular interest to AppEngine developers.

Nice!

dateutil 1.4 is out

Published
by
Gustavo Niemeyer
on 2008-03-03
in Project and Python
. 2 Comments

Friday I’ve released version 1.4 of dateutil. There are some interesting fixes there, so please upgrade if you have the chance.

Enhancements on geohash.org

Published
by
Gustavo Niemeyer
on 2008-03-01
in GPS, Project and Python
. 0 Comments

Some improvements to geohash.org were made. Some of them were
motivated by a conversation with Rodrigo Stulzer.

  • Support for geocoding addresses (city names, whatever). E.g. http://geohash.org/?q=21 Millbank, London
  • Support for moving the Geohash marker in the embedded map, so that modifying the position visually is easier.
  • Support for providing a “name” to Geohashes, by appending a colon and the name, in a nice format. E.g. http://geohash.org/c216ne:Mt_Hood
  • Provided a bookmark to get a Geohash while in Google Maps.
  • Provided a Google Maps Mapplet. When enabled, it adds a Geohash marker identifying the Geohash position in Google Maps, and it may be moved around. Here is a screenshot:

Check out the Tips & Tricks page for details on these features.

geohash.org is public!

Published
by
Gustavo Niemeyer
on 2008-02-26
in Article, GPS, Project and Python
. 7 Comments

After about one year writing this service in my spare time, it’s finally out.

geohash.org offers short URLs which encode a latitude/longitude pair, so that referencing them in emails, forums, and websites is more convenient.

Geohashes offer properties like arbitrary precision, similar prefixes for nearby positions, and the possibility of gradually removing characters from the end of the code to reduce its size (and gradually lose precision). I’ve put the algorithm created in the public domain. Some details may be seen in the Wikipedia article about it (hopefully that’ll help establishing prior art, and prevent Microsoft from patenting it).

To obtain the Geohash, the user provides latitude and longitude coordinates in a single input box (most commonly used formats for latitude and longitude pairs are accepted), and performs the request.

Besides showing the latitude and longitude corresponding to the given Geohash, users who navigate to a Geohash at geohash.org are also presented with an embedded map, and may download a GPX file, or transfer the waypoint directly to certain GPS receivers. Links are also provided to external sites that may provide further details around the specified location.

Mocker 0.10 and trivial patch-mocking of existing objects

Published
by
Gustavo Niemeyer
on 2007-12-09
in Project, Python, Snippet and Test
. 2 Comments

Mocker 0.10 is out, with a number of improvements!

While we’re talking about Mocker, here is another interesting use case, exploring a pretty unique feature it offers.

Suppose we want to test that a method hello() on an object will call self.show(”Hello world!”) at some point. Let’s say that the code we want to test is this:

 class Greeting(object):

     def show(self, sentence):
         print sentence

     def hello(self):
         self.show("Hello world!")

This is the entire test method:

def test_hello(self):
    # Define expectation.
    mock = self.mocker.patch(Greeting)
    mock.show("Hello world!")
    self.mocker.replay()

    # Rock on!
    Greeting().hello()

This has helped me in practice a few times already, when testing some involved situations.

Note that you can also passthrough the call. In other words, the call may actually be made on the real method, and mocker will just assert that the call was really made, whatever the effect is.

One more important point: mocker ensures that the real method exists in the real object, and has a specification compatible with the call made. If it doesn’t, and assertion error is raised in the test with a nice error message.

UPDATE: The method for doing this is actually mocker.patch() rather than mocker.mock(), as documented. Apologies.

Partial stubbing of os.path.isfile() with Mocker

Published
by
Gustavo Niemeyer
on 2007-11-22
in Project, Python, Snippet and Test
. 0 Comments

One neat feature which Mocker offers is the ability to very easily implement custom behavior on specific functions or methods.

Take for instance the case where you want to pretend to some code that a given file exists, but you don’t want to get on the way of everything else which needs the same function: 

>>> from mocker import *
>>> mocker = Mocker()
>>> isfile = mocker.replace("os.path.isfile", count=False)
>>> _ = expect(isfile("/non/existent")).result(True)
>>> _ = expect(isfile(ANY)).passthrough()

>>> mocker.replay()

>>> import os
>>> os.path.isfile("/non/existent")
True
>>> os.path.isfile("/etc/passwd")
True
>>> os.path.isfile("/other")
False

>>> mocker.restore()

>>> os.path.isfile("/non/existent")
False

Notice that the count=False parameter is available in version 0.9.2. Without it Mocker will act in a more mocking-strict way and enforce that the given expressions should be executed precisely the given number of times (which defaults to one, and may be modified with the count() method).

More releases: dateutil 1.3 and nicefloat 1.1

Published
by
Gustavo Niemeyer
on 2007-11-19
in Project and Python
. 0 Comments

A couple of additional releases tonight: dateutil 1.3, and nicefloat 1.1.

They’re both bug fixing releases.

Mocker 0.9

Published
by
Gustavo Niemeyer
on 2007-11-17
in Project and Python
. 6 Comments

A few more improvements were made to Mocker.

Storm has always reused connections (connection pooling?)

Published
by
Gustavo Niemeyer
on 2007-11-17
in Project and Python
. 1 Comment

I’ve recently seen some comments here and there about the lack of connection pooling as an argument for Storm to be faster, and that once this is supported it will be slower, or even as a reason for people not to use Storm at all.

So, let me kill this argument here, at once.

We have not developed Storm only for toy projects that take 10 connections a day. We have developed Storm for heavy duty web sites like Landscape and Launchpad, and we’re proud to see it being used not only in our systems, but also out there in the wild, like for instance in large scale sites developed by the fantastic guys at Lovely Systems.

So how does the connection reuse work in practice, you ask. Here is how:

In Storm, the database is abstracted behind a small, simple, and flexible API, offered in the Store class. You use an instance of this class to deal with objects coming from a given database, and this instance will handle several aspects of your interaction with the database, such as committing, rolling back, caching, ensuring that a given row in the database maps to a single instance in memory, control of dirty objects, flushing, and so on. Pretty much all of these aspects require a correct transactional behavior to work well, and in practice this means we’ve decided that to maintain the API nice and clean, each Store is internally associated with a single Connection object. You can have as many stores as you want, connecting to the same database or to different ones, and using the same model class or entirely different code bases.

So, to summarize the above paragraph, a simple Store instance is your portal to the database. You need one of these instances around to add objects to the database (Storm won’t guess which Store you want to add things to), and to retrieve objects from it.

Considering that, if you want to reuse a connection, it’s very simple: keep your Store instance around. That’s even a strange advice, since you’re already doing that if you’re using Storm in the first place. The code in trunk, which is about to be released as version 0.12, even handles reconnections for you gracefully, including correct transactional behavior.

We even offer a tool that deals with more advanced Store management in a very comfortable way for Zope 3. In the future, we’re likely to offer the same kind of facility in a more generic API.

So, connection reuse is there, and we have always benefited from it. Connection pooling? No, thanks. We’re doing very well without the complexity and overhead.