Efficient algorithm for expanding circular buffers

Circular buffers are based on an algorithm well known by any developer who’s got past the “Hello world!” days. They offer a number of key characteristics with wide applicability such as constant and efficient memory use, efficient FIFO semantics, etc.

One feature which is not always desired, though, it the fact that circular buffers traditionally will either overwrite the last element, or raise an overflow error, since they are generally implemented as a buffer of constant size. This is an unwanted property when one is attempting to consume items from the buffer and it is not an option to blindly drop items, for instance.

This post presents an efficient (and potentially novel) algorithm for implementing circular buffers which preserves most of the key aspects of the traditional version, while also supporting dynamic expansion when the buffer would otherwise have its oldest entry overwritten. It’s not clear if the described approach is novel or not (most of my novel ideas seem to have been written down 40 years ago), so I’ll publish it below and let you decide.

Continue reading

Introducing The Hacking Sandbox

When I started programming in Python long ago, one of the features which really hooked me up was the quality interactive interpreter offered with the language implementation. It was (and still is) a fantastic way to experiment with syntax, semantics, modules, and whatnot. So much so that many first-class Python practitioners will happily tell you that the interactive interpreter is used not only as a programming sandbox, but many times as the their personal calculator too. This kind of interactive interpreter is also known as a REPL, standing for Read Eval Print Loop, and many languages have pretty advanced choices in that area by now.

After much rejoice with Python’s REPL, though, and as a normal human being, I’ve started wishing for more. The problem has a few different levels, which are easy to understand.

Continue reading

The forgotten art of error checking

I was just rambling randomly yesterday, in the usual microblogging platforms, about how result checking seems to be ignored or done badly. The precise wording was:

It’s really amazing how little attention error handling receives in most software development. Even *tutorials* often ignore it.

It indeed does amaze me. It sometimes feels like we write code for theoretical perfect worlds.. “If the processor executes exactly in this order, and the weather is calm, this program will work.”. There are countless examples of bad assumptions.. someday I will come with some statistics of the form “Every N seconds someone forgets to check the result of write().”.

Continue reading

Using pyperl to build a python-like xchat interface for perl

My weekend was taken by boring academic stuff. More specifically, I’ve built a validator in C, including a tokenizer, a parser, and a scope system, to check for a small subset of the C language syntax. Anyway.. I was still able to do something interesting after all. :-)

Taking further my research about pyperl, I’ve wondered if it would be possible to wrap the XchatPython plugin interface for perl usage. Actually, it turned out that besides being possible, it was easy. It was just a matter of putting the following wrapper in the ~/.xchat2 directory:

import xchat
import perl
perl.eval("$xchat = Python::eval('xchat')")
perl.eval("do '" + __file__[:-3] + ".pyl'")
__module_name__ = perl.eval("$__module_name__")
__module_version__ = perl.eval("$__module_version__")
__module_description__ = perl.eval("$__module_description__")

This wrapper will load a file with the same name as itself, but with a .pyl extension, which should contain perl code defining the plugin (I haven’t used the .pl extension because this would conflict with the internal perl plugin of xchat). That code may access the $xchat object to interface with xchat, using the same API as defined in XchatPython.

Here is a small sample module in perl, defining the /test command:

#!/usr/bin/perl
$__module_name__ = "perlwrap";
$__module_version__ = "1.0";
$__module_description__ = "Testing perlwrap.";

sub test {
    $xchat->prnt("Test command!");
    return $xchat->EAT_ALL;
}

$xchat->hook_command("test", &test);

$xchat->prnt("Plugin perlwrap loaded!n");

Notice that the interface is exactly the same, including the meta-information.

Unfortunately, there seems to exist some bug in the MULTI_PERL support of pyperl which makes it lose track of function references once the python support commutes the thread states. To make this trick work, remove the MULTI_PERL support by deleting the MULTI_PERL file in the root of the pyperl distribution.

Webmin and Perl inside Python

Today I’ve been talking to Cavassin about Webmin, and wondering if it would be possible to write Python modules for it. I was sure that I wasn’t being original in that idea, and Cavassin also mentioned that he had read something about it somewhere, so I’ve looked in google for possible references.

Of course, we were right. I have found the python-webmin project. OTOH, I’ve looked in the code, and that’s not really what I had mind. This module reimplements the Webmin API in Python. This means that one will have to maintain it forever, and check for updates in every Webmin release, and even then, it won’t be able to do more advanced stuff which requires deeper interaction with the Webmin system.

The correct solution for that would be to somehow wrap the Webmin API in a way that Python calls the real methods, thus requiring little or no maintenance. It’d be even better if this system was generic and not bound to Webmin in any way. While looking for the solution, I found Inline, which did exactly what I had in mind, but reversed (Python inside Perl), and PyInline, which unfortunately doesn’t seem to support Perl yet. Then, I finally found what I was looking for: pyperl.

The pyperl project embeds a Perl interpreter inside a Python module, allowing arbitrary Perl functionality to be used from Python. It’s really amazing to see it working. Have a look at these simple examples:

>>> import perl
>>> perl.eval('sub func { printf("Hello $_[0]!n"); }')
>>> ret = perl.call("func", "world")
Hello world!

>>> func = perl.get_ref("func")
>>> ret = func("world")
Hello world!

>>> perl.require("Digest::MD5")
1
>>> md5 = perl.callm("new", "Digest::MD5")
>>> md5.add("something").hexdigest()
'437b930db84b8079c2dd804a71936b5f'

Isn’t it fantastic? :-)

Another amazing thing is that pyperl also includes support for doing Python inside Perl. Of course, you can do Python inside Perl inside Python:

>>> n = 5
>>> perl.eval('Python::eval("n")*2')
10

Now, it’s just a matter of finding some time to create a basic wrapper on top of the Webmin Perl API, turning it into something more pythonic and also protecting Webmin modules developed in Python from changes in the pyperl API.

References