Archive for the 'C/C++' Category

Comparing package versions in PostgreSQL

Published
by
Gustavo Niemeyer
on 2006-07-30
in C/C++, PostgreSQL, Project and Python
. 1 Comment

This weekend I’ve played a bit with PostgreSQL extensions written in C.

A while ago I wrote a Python C extension for Smart to compare Debian package versions. Now I was trying to do something similar inside PostgreSQL, and thus ported the original Python C extension code to a PostgreSQL C extension. It enables queries like the following:

# SELECT 'Matched' WHERE deb_version_match('1.2', '<=', '2.0');
  ?column?
----------
 Matched

The implementation of the PostgreSQL C extension was quite straightforward, but I’m a bit disappointed by the performance of PL/PGSQL.

I’ve made tests using two environments. One of them is a PL/Python function executed as a trigger, which calls the original Python C extension and executes SQL back in PostgreSQL using the plpy module. The other is a PL/PGSQL function which uses the implemented PostgreSQL C extension directly.

Considering that the function logic consisted of one loop over a SELECT statement, a few tests, and an INSERT statement, I was expecting that the overhead introduced by going back and forth between the PostgreSQL state and the Python interpreter state would be a lot more noticeable when compared with PL/PGSQL executing an internal PostgreSQL function. Tests have shown about 10% of improvement roughly, when doing a similar logic over about 5000 items.

I’m not yet sure if the speed improvement pays off the limited debugging feedback provided by the PL/PGSQL interpreter on errors.

Smart Package Manager is out!

Published
by
Gustavo Niemeyer
on 2004-12-09
in C/C++, Project and Python
. 0 Comments

After 6 months of fun working on the project in silence, Smart Package Manager was finally released.

As the README says:

The Smart Package Manager project has the ambitious objective of creating smart and portable algorithms for solving adequately the problem of managing software upgrading and installation. This tool works in all major distributions, and will bring notable advantages over native tools currently in use (APT, APT-RPM, YUM, URPMI, etc).

Some of the interesting features, which are covered in more detail in the README file:

  • Smart transactions
  • Multiple interfaces
  • Support for several channel formats
  • Priority handling
  • Autobalancing mirror system
  • Parallel downloading mechanism
  • Flexible removable media support

Have fun! ;)

Embedding Lua interpreter into RPM

Published
by
Gustavo Niemeyer
on 2004-03-23
in C/C++, Lua and Patch
. 0 Comments

I’ve recently committed into the RPM CVS HEAD the internal support for Lua scripts.

Please, notice that this is experimental stuff.

Why embedding Lua in RPM?

  • Many scripts execute simple operations which in an internal interpreter require no forking at all
  • Internal scripts reduce or eliminate external dependencies related to script slots
  • Internal scripts operate even under unfriendly situations like stripped chroots (anyone said installers?)
  • Internal scripts in Lua are really fast
  • Syntax errors in internal scripts are detected at package building time

How it works?

Just use -p <lua> in any script slot (%pre, %post, etc).

For example: 

%pre -p <lua>
print("Wow! It really works!")

What is accessible from Lua?

The standard Lua library, the posix module (basic system access, by Luiz Henrique de Figueiredo and Claudio Terra), and the rex module (regular expressions, by Reuben Thomas).

Macro support

Support for Lua macros was also introduced. It means that one can create custom content using Lua macros anywhere.

For example: 

%{lua: print("Requires: hello-world > 1.0") }

More additions to APT-RPM Lua interface

Published
by
Gustavo Niemeyer
on 2004-03-14
in C/C++, Lua and Project
. 0 Comments

The APT-RPM Lua interface is constantly being improved. This time, the following functions were added:

pkgid() and verid()

Return a unique integer identifying a package or a version.

verpkg()

Returns the parent package of some given version.

verdeplist()

Returns a list of dependencies for a given package, including complete information about it.

These new functions were introduced to give support for something which is frequently asked by APT-RPM users: the ability to discover which installed packages are not required by any other installed package.

Here is a script using these functions to list these packages. This script will be called list-nodeps, and will be available in the contrib/ directory of the next APT-RPM release. 

-- Collect dependencies from installed packages
deplist = {}
verlist = {}
for i, pkg in ipairs(pkglist()) do
    ver = pkgvercur(pkg)
    if ver then
        table.insert(verlist, ver)
        for i, dep in ipairs(verdeplist(ver)) do
            for i, depver in ipairs(dep.verlist) do
                deplist[verid(depver)] = true
            end
        end
    end
end

– Now list all versions which are not dependencies
for i, ver in ipairs(verlist) do
    if not deplist[verid(ver)] then
        name = pkgname(verpkg(ver))
        if name ~= “gpg-pubkey” then
            — Print package name and version without epoch
            — (rpm -e friendly ;-).
            print(name..”-”..string.gsub(verstr(ver), “^%d+:”, “”))
        end
    end
end

More information about the introduced functions is available in https://moin.conectiva.com.br/AptRpm/Scripting

New KDE frontend for APT-RPM

Published
by
Gustavo Niemeyer
on 2004-03-11
in C/C++ and Project
. 1 Comment

Here is a screenshot of Kynaptic, a new experiment I’m working on. This is a KDE-based frontend for APT-RPM and APT. Kynaptic is based on Synaptic, and shares the same base code on top of the APT library.

On the right side you see a new toy: the interactive search dialog. This should definetly improve the user experience while looking for packages, and could eventually be ported to Synaptic itself.

Another interesting point is that during the development process of Kynaptic I’m also cleaning up the Synaptic code, something I had in my mind for some time already.

Toy interpreter for Linear Algebra

Published
by
Gustavo Niemeyer
on 2004-03-02
in C/C++, Math and Project
. 1 Comment

While working with Linear Algebra, I’ve decided to build a toy interpreter in C to play around. This was a quite interesting experiment for myself, since it was the first time I’ve built a complete (with tokenizer, compiler, and interpreter) and auto-suficient (no external dependencies, no additional tools) interpreter from the ground.

The interpreter is currently 1076 lines long, and is based on a tokenizer for the grammar, a compiler which creates a list of trees of expressions, an interpreter for the generated structure, and two modules for symbol maintenance and matrix operations.

Here is a quick example: 

a = 2*3+4/2
print(a)

b = [1,2,3|4,5,6]
c = [1,2,3|4,5,6]
print((b+c)/2)
print(trans(b))

d = [1,0|-2,3|5,4|0,1]
e = [0,6,1|3,8,-2]
print(d*e)

f = [1,2|3,4]
print(det(f))
f = [2,3,4|-5,5,6|7,8,9]
print(det(f))
f = [1,-4,2,-2|4,7,-3,5|3,0,8,0|-5,-1,6,9]
print(det(f))

And here is the output:

8
[ 1, 2, 3 | 4, 5, 6 ]
[ 1, 4 | 2, 5 | 3, 6 ]
[ 0, 6, 1 | 9, 12, -8 | 12, 62, -3 | 3, 8, -2 ]
-2
-45
2042

New RPM transaction locking

Published
by
Gustavo Niemeyer
on 2004-03-01
in C/C++ and Patch
. 0 Comments

Last week I’ve worked with Jeff Johnson to fix a small RPM issue which was bothering us for quite some time. Until this change, RPM was accepting two transactions to be run at the same time but, unfortunately, this might put the database in an inconsistent state, and some headers might get lost while restoring it with –rebuilddb.

Our fix was rather simple, based on fcntl() locking while transactions are being committed to the system. This should be enough, protecting databases while the issue is not fixed The Right Way (the time consuming one ;-) ).

Developers of applications linked with RPM should not worry about additional errors, since the current behavior is to wait for the lock to be released while entering in the transaction committing function.

Compiler warnings in Python’s SRE

Published
by
Gustavo Niemeyer
on 2004-02-17
in C/C++, Patch and Python
. 0 Comments

After a long time, and many people complaining about it, I finally took some time to fix some annoying compiler warnings in Python’s regular expression engine. Since it’s a rather uncommon case, I’ll explain it here with a quick example.

Have a look at the following code: 

#include <stdio.h>

int main(int argc, char *argv[])
{
   unsigned char c = 1;
   if (c < 256)
      printf("Hello world!\n");
   return 0;
}

If you try to compile that code, you’ll probably get a warning like this: 

test.c: In function `main':
test.c:7: warning: comparison is always true due to
limited range of data type

Yes, the compiler is right. Our char type will never reach the given limit. On the other hand, suppose that this code is preprocessed, and that the c variable has sometimes a multibyte character type, like wchar_t, for example. In this case our test is legitimate, and the dozens of warnings being caused by a common macro are really annoying.

There are many different ways to remove these warnings. Unfortunately, the most obvious one, which is casting the variable to a larger type, doesn’t work as expected.

My adopted solution was to reimplement the same test in another way, surpassing the compiler warnings for this specific case. Instead of writing (c < 255), I’ve written (!(c & ~255)). This is 100% safe for any unsigned type, which is my case. Ok, perhaps it’s a little bit sick, but a large comment should leave everyone aware about the rationale, and away from the warnings.

Lunatic Python

Published
by
Gustavo Niemeyer
on 2003-12-14
in C/C++, Lua, Project and Python
. 0 Comments

Lunatic Python is a two-way bridge between Python and Lua, allowing these languages to intercommunicate. Being two-way means that it allows Lua inside Python, Python inside Lua, Lua inside Python inside Lua, Python inside Lua inside Python, and so on.

Here are two examples giving an idea about what it does.

Lua inside Python

>>> table = lua.eval("table")
>>> def show(key, value):
...   print "key is %s and value is %s" % (`key`, `value`)
...
>>> t = lua.eval("{a=1, b=2, c=3}")
>>> table.foreach(t, show)
key is 'a' and value is 1
key is 'c' and value is 3
key is 'b' and value is 2
>>>

Python inside Lua 

> function notthree(num)
>>   return (num ~= 3)
>> end
> l = python.eval("[1, 2, 3, 4, 5]“)
> filter = python.eval(”filter”)
> =filter(notthree, l)
[1, 2, 4, 5]

Updated XChat’s python plugin

Published
by
Gustavo Niemeyer
on 2003-10-20
in C/C++, Project and Python
. 0 Comments

After a long time of inactivity, I’ve updated the python plugin of xchat. There were some pending bugs, and new xchat features to catch. Here is the list of changes:

  • Fixed the reentrance problem which made xchat freeze when some python code made xchat trigger a python callback somewhere.
  • Implemented xchat_emit_print() and xchat_get_prefs() support.
  • Fixed softspace support for python 2.3. This problem was making print statements yield an extra space at the next line start.
  • Now in the python console, every input line is printed after >>> in the query window, making it more convenient and familiar.
  • Other minor changes.

The documentation in http://labix.org/xchat-python has been updated as well. For current users of xchat 2.0.5, the updated file is in the Download section.