Jesses!

Static LXML builds

2009-02-12T14:36:00.005+01:00

I've built a recipe for building LXML on OSX boxes statically -- w/o dependencies to libxslt and libxml2.

http://pypi.python.org/pypi/z3c.recipe.staticlxml

This works for quite some people -- please report back if it does'nt work for you (and if it works).

Geek Music

2008-06-06T19:13:00.001+02:00

Big Ideas (Don't get any) from 1030 on Vimeo.

lxml on OSX -- solved

2008-06-05T21:07:00.004+02:00

Hi guys. I've been investigating building lxml on OSX for quite some time now.

Here's the scoop so far:

I want to use buildout
I want NOT to use Apple's libraries (libxml, libxslt)
On OSX there's no "--rpath" option available, thus there's no way to tell an executable on link time where to look for dynamic libraries
This is disgusting (do we have 2008 or what?)

Ok, that's what I've come up with. Note the z3c.recipe.egg, it allows for arguments to the setup script, which the original zc.recipe.egg does not. The version 2.0.6 of lxml has a fix for OSX in, which allows to do away with DYLD_LIBRARY_PATH, BUT you need a DYLD_FORCE_FLAT_NAMESPACE=1. Don't ask. I have no idea.

[lxml]
parts=
libxml2
libxslt
lxml-download
lxml-build

[libxml2]
recipe = zc.recipe.cmmi
url = http://ftp.gnome.org/pub/GNOME/sources/libxml2/2.6/libxml2-2.6.26.tar.gz
extra_options = --without-python

[libxslt]
recipe = zc.recipe.cmmi
url = http://ftp.gnome.org/pub/GNOME/sources/libxslt/1.1/libxslt-1.1.16.tar.bz2
extra_options = --with-libxml-prefix=${buildout:directory}/parts/libxml2/
            --without-python

[lxml-download]
recipe = plone.recipe.distros
urls = http://codespeak.net/lxml/lxml-2.0.6.tgz
version-suffix-packages = lxml-2.0.6.tgz

[lxml-build]
recipe = z3c.recipe.egg:setup
setup = ${lxml-download:location}/lxml
args =
clean
build
bdist_egg
--with-xslt-config=${buildout:directory}/parts/libxslt/bin/xslt-config
develop = true

You use it like:


[buildout]
extends=
   lxml.cfg
parts=
   ${lxml:parts}
eggs=
   ${lxml:eggs}

lxml on OSX -- again

2008-05-19T20:22:00.009+02:00

Hi folks :)

Today, I've again tried to install lxml on OS X. For a project I do for a customer (migrating featurecreep CMS sites to Plone), I use the lxml parser. I get regular segfaults, this time using python 2.4 (I use macport for python 2.4).
I investigated again, and came along this post. The scoop is:

after blood, sweat, and some tears (kidding) this is *all* I needed, it seems:

export CFLAGS="-flat_namespace"

...no static builds libxml2 libs, no buildout recipe.  I just set that and ran:

python setup.py bdist_egg
--with-xml2-config=/opt/local/bin/xml2-config
--with-xslt-config=/opt/local/bin/xslt-config

which uses the libxml2 and etc. installed by ports.  In fact, as long
as /opt/local/bin is on my path that should work without having to set
paths (i.e. from easy_install).  All my tests that were segfaulting
are now passing.  This appears to be the exact same behavior I got by
setting DYLD_FORCE_FLAT_NAMESPACE at runtime but without the side
affect of applying itself to anything else running in my shell ;)

so, I'm thinking this is just two lines of code added to cflags() ...

if sys.platform in ('darwin',):
result.append('-flat_namespace')

Well. What can I say. Interesting poinst so far for OS X development:

no -rpath supported
no static libraries (Yup! No joke)

Read This Post!

2008-05-18T12:33:00.000+02:00

I found this post quite educating.

Aardvark bookmarklet

2008-05-15T23:02:00.000+02:00

Duh.

One should read on, THEN post.

Aardvark for Safari --- sort of

2008-05-15T22:57:00.003+02:00

Well, this might be old news for you, but by dragging the "run demo" link on the Aardvark home page to the Safari bookmark bar actually allows you to inject Aardvark to the page your're currently viewing.

I think that's cool and useful :)

incredible stuff

2008-05-15T22:25:00.008+02:00

I've just found this incredible stuff on the web. I thought I just might blog about it.

git shelves

import gitshelve

data = gitshelve.open(repository = '/tmp/data.git')

data[key] = "Hello, world!"
Data.Sync()
del data[key]
data.close()

How cool is this? Distributed CMS someone?

Original post here. Git it here,
processing on **javascript**

This guy actually ported the processing language to Javascript.

Thats f***ing awesome!

LXML on OS X (again)

2008-05-09T01:08:00.003+02:00

It seems that building lxml on Mac OS X is still a problem. I'm trying to use buildout as in:

[libxslt]
recipe = zc.recipe.cmmi
url = http://ftp.gnome.org/pub/GNOME/sources/libxslt/1.1/libxslt-1.1.16.tar.bz2
extra_options = --with-libxml-prefix=${buildout:directory}/parts/libxml2/
              --without-python

[lxml]
recipe = zc.recipe.egg:custom
egg = lxml
include-dirs = ${buildout:directory}/parts/libxml2/include/libxml2
             ${buildout:directory}/parts/libxslt/include
library-dirs = ${buildout:directory}/parts/libxml2/lib
             ${buildout:directory}/parts/libxslt/lib
rpath = ${buildout:directory}/parts/libxml2/lib
      ${buildout:directory}/parts/libxslt/lib

That fails when buildout tries to build the lxml egg. Looking closely at the error, I discovered that despite pointing it at the libxslt built, it still uses the one found in /usr/lib.

To be more specific: lxml tries to find xslt-config, which is in $BO/parts/libxslt/bin/xslt-config, but of course finds the one in /usr/bin first.

What I'd like to do is to add something like:


# this would be needed but zc.recipe.egg:custom doesn't allow for that
# Doh.
extra_options= --with-xslt-config=${buildout:directory}/parts/libxslt/bin/xslt-config

That option is available in zc.recipe.cmmi, but not in zc.recipe.egg. Looking at custom.py there I sadly found no obvious way to add this as it uses some version of easy_install.

D'oh.

Possible solutions:

build egg manually, add to egg cache (yuck-bleah)
build manually for OS X, add to own egg server (yuck)
roll-my-own recipe (work)

Use git -- have more fun.

2008-05-08T10:50:00.003+02:00

Following this post, I was lured into trying the git svn backend. We're using SVN at our office for hysterical reasons and this seemed a good way to use git locally and svn as a backend.

So basically what you do is:

$ mkdir somedir
$ cd somedir
$ git-svn init -s $svnurl

That will initialize the git repository. The -s tells git-svn to assume the standard directory scheme for tags and branches, that is, below the svnurl there should be a "tags", "branches", "releases". Now fetch the repo:

$ git-svn fetch

That'll suck all svn revisions to your local git repo.
Now hack away and use the usual git commands to check in locally.

Use something like gitNub to view the repository changes.

Enjoy :)

git +ssh on a mac using macports

2008-05-07T22:09:00.004+02:00

To get git installed with subversion support using macports you need to do the following:

$ sudo port install git-core +ssh

Note the +ssh.

Yo! Back from NYC vudo sprint.

2008-05-07T12:02:00.004+02:00

I'm still a bit exhausted from the coding marathon we did at the NYC vudo sprint.

We had a great time and I enjoyed pair programming with Malthe Borch very much.

We did some sort of ZODB abstraction using SQLAlchemy called z3c.dobbin. That is, z3c.dobbin uses the z3 schemata (interfaces!) to transparently create SQL tables. Each table will map to one single interface, and dobbin will join these tables based on a dynamically created mapper. That means, that every interface is available separately for doing SQL queries. This should allow for pretty fast and intuitive queries.

We're going to use a relatively new approach to editing web pages -- it'll be more page centric in contrast to being content type centric. There simply will not be any content types in vudo.

Also the skinning will be completely different -- we'll use the designed mock-ups directly, and we'll server them unaltered. That compares a little to deliverance, but only a little. We'll call these mock ups "layouts" and each page in vudo is associated with a lay out. These layouts define "regions", where vudo inserts rendered content at run time. All this is configured using a ZCML directive -- that is w/o touching the mock up, and w/o having the designer be forced to do special markup (e.g. using class prefixes or id prefixes).

I think that's cool :)

And I can't wait to replace this crappy blogging interface with vudo ...

iPhone Developer Program

2008-05-07T11:57:00.003+02:00

Yes! I've got a mail from Apple that I'm accepted to the iPhone developer program!

I hope I can make my ideas become real -- I want to have a native front end on my iPhone

to the web framework I'm going to use Really Soon Now (TM).

Time passed ....

2008-04-22T21:54:00.004+02:00

.... and things happened:

I was at the Plone Conference '07
I was at this year's Snow Sprint

And most exiting to me, I'll visit NYC the first time in my life to get hands on some really cool stuff.

I'm really into python and web programming now, leaving my old work behind.

Other than that, I've started (again) to work for zest to help paying bills.

Ahhhh, new post at last.

2007-04-25T17:51:00.000+02:00

Wohoo. A new post.

Finally we sorted out some issues at my company, where I'm partner.

And I got more into Python and Plone. I visited some Plone sprints:

I'll try to attend some more, like the sprint in Potsdam later this year.

Oh, yes, the Plone Conference and Sprint? I'll be there!

Divmod.org AXIOM Ressources.

2006-11-01T16:29:00.000+01:00

What is Axiom?

Some quotes:
"Axiom's goal is to provide an efficient, non-SQL interface to an existing relational database." here

Why ise it?

Axiom is fast.

The question for me is, how does Axiom differ from SQLObject, SQLAlchemy or TurboEntity?

AJAX links

2006-11-01T15:47:00.000+01:00

While surfing the web I just found these ressources about AJAX and Mochikit.

Heres the website of "The Ajax Exprience" event.

Here are some slides for an Mochikt tutorial. These slides are written for S5 using rst2s5 from python docutils. Need to investigate!

Agile Development

2006-11-01T15:40:00.000+01:00

In my current search for a development model which works better when working in a team I have these links to share:

A review of a the book "Agile Estimating and Planning" by Marc Cohen here at isotoma

A list of articles here at the Agile Alliance website.

p2p -> q2q

2006-10-31T01:19:00.000+01:00

I've been playing around with divmod.org quotiont 2 quotient protocol implementation called vertex.

The q2q protocol is a p2p protocol where you can basically connect to a thing called a "resource".

The Protocol offers resource look up (Where is X?) and authentication (Is it X I'm talking to). The most interesting bit for me is that they implemented STUN and NAT traversal.

Vertex is a demo and proof-of-concept type implementation of this protocol which uses this protocol to authenticate users and send files through the connection.

Success!

2006-10-25T00:59:00.000+02:00

Yeee-haaa!

I've just connected to Skype using Python and the Skype API Framework. I get notifications, and can send commands:


seletz@yeti:~/develop/python/pyskype/test $python skypeapi-test-osx.py
2006-10-25 01:05:55.934 Python[14094] running event loop
2006-10-25 01:05:56.935 Python[14094] connecting to skype
2006-10-25 01:05:56.941 Python[14094] waiting ....
2006-10-25 01:06:00.547 Python[14094] skypeAttachResponse: 1
2006-10-25 01:06:00.576 Python[14094] skypeNotificationReceived: CONNSTATUS ONLINE
2006-10-25 01:06:00.578 Python[14094] skypeNotificationReceived: CURRENTUSERHANDLE seletz29
2006-10-25 01:06:00.578 Python[14094] skypeNotificationReceived: USERSTATUS ONLINE
2006-10-25 01:06:06.942 Python[14094] sending
2006-10-25 01:06:06.942 Python[14094] got answer: None
2006-10-25 01:06:06.943 Python[14094] skypeNotificationReceived: PROTOCOL 1

I've started a new project at Google Code, where I try to create a platform-independent Python wrapper for the Skype API.

The trick was, of course, RTFM. I just needed to use PyObjC to create a informal protocol for the delegate class I use to connect with the Skype API. It's rather byzantine how Apple encodes type information.

WiKi entry on Skype Devzone

2006-10-24T11:27:00.000+02:00

I've created a WiKi entry over at the Skype Devzone WiKi about code snippets. Unfortunately, you need a Skype Account to use that WiKi. Doh.

I plan to investigate all available OS Versions of the Skype API and build a common Python wrapper for it.

Skype API on OS X

2006-10-23T22:43:00.000+02:00

Well here's my code so far. It uses pyobjc to fetch the Skype Public API framework and then defines a class to be used as a delegate for the Skype API. Skype is supposed to call the skypeAttachResponse() callback, but that does not happen.

BTW, note the magic in the objc.loadBundle() call ....

Sending messages to the Skype API then results in a log message stating that no client is attached. Doh.

Calling isSkypeRunning() et al works, so I'm actually pretty sure that I'm talking to the Skype API ;)

Any pointers would be greatly appreciated ;)

Update: I noticed that the signature of the callback methods of my Python Delegator class is probably not correct. I'll investigate further. There are pointers to that inside the PyObjC docs.


import objc
from Foundation import *

objc.loadBundle( "SkypeAPI", globals(),
bundle_path=objc.pathForFramework( \
"~/Library/Frameworks/Skype.framework" ) )

#objc.setSignatureForSelector( "SkypeAPI",
#   "sendSkypeCommand:", "v@N^@:@" )
SkypeAPIDelegate = objc.protocolNamed( "SkypeAPIDelegate" )

class Delegate(NSObject, SkypeAPIDelegate):
def init(self, app_name):
  self = super( Delegate, self).init()
  if self is None:
      return None

  self._appname = app_name

  return self

def clientApplicationName( self ):
  return self._appname

def skypeAttachResponse( self, code ):
  NSLog( "skypeAttachResponse: %d" % ( code ) )

def skypeNotificationReceived( self, nsstring ):
  NSLog( "skypeNotificationReceivest: %s" % ( nsstring ) )

def skypeBecameAvailable( self, notification ):
  NSLog( "skypeBecameAvailable: %s" % ( notification ) )

def skypeBecameUnavailable( self, notification ):
  NSLog( "skypeBecameUnavailable: %s" % ( notification ) )

def do_connect():
delegate = Delegate.alloc().init( "testapp" )

NSLog( "connecting to skype" )
SkypeAPI.setSkypeDelegate_( delegate )
SkypeAPI.connect()

NSLog( "sending" )
SkypeAPI.sendSkypeCommand_( u"PROTOCOL 1" )

def main():
import sys
from PyObjCTools import AppHelper

NSLog( "running event loop" )
AppHelper.callLater( 5, do_connect )
AppHelper.runConsoleEventLoop( installInterrupt=True )

if __name__ == "__main__":
main()

# vim: set ts=4 sw=4 expandtab :

Note: I love Vim ;)

Embedding Python and Multithreading

2006-10-23T22:27:00.000+02:00

Just found this post on comp.lang.python about an article detailing the hoops one needs to jump through when it comes to embedding python and you have a multithreaded application.

Note that all that still means that you can't have one Python interpreter running in multiple threads. If I understood things as they are, Python currently needs a 1:1 relationship between Python interpreters and threads. That is, you can have more than one Python interpreter per process but only one per thread. You can have more than one interpreter per process if you really want, but these threads then compete against the same GIL.

This is due to the fact that the standard CPython implementation uses a so-called global interpreter lock (GIL) to serialize access to the internal Python API. There's one GIL per interpreter.

Also note that there's hidden pitfall here: The interpreter periodically (every n byte code instructions) yields execution and releases the GIL. That means if other threads pend on the GIL they might get runnable, which is perhaps not what one might expect.

Calling C extension is another common way to release the GIL, such that a expensive calculation in does not block other threads.

Python Skype API

2006-10-21T22:36:00.000+02:00

Hrmph. My Python wrapper using the ctypes module which comes with Python 2.5 segfaults. I can check if Skype is running, using the IsSkypeRunning() call, but as soon as I try to use SetSkypeDelegate() it segfaults. Now I'm downloading ctypes 1.0.1, perhaps there's more docs and examples there. Especially on how to use the Structure class with Python callbacks.

Update: No avail. Either SF is down or I don't know what. I'm unable to download the ctypes stuff.

Skype API

2006-10-21T17:42:00.000+02:00

I just discovered the Skype API. I know, Skype is not open-source, but hey ...

The API is quite cool. Apart from the pretty obvious API access to the voice call and chat stuff, there's an API for transparently sending raw streams between programs. So, one could use the Skype API to use a pretty nifty transparent channel to send data between applications. They call this the Application2Application Protocol.

This could be the last building block I need for a pet project I have, the Personal Information Server. There I need a method to communicate with different PIS in a P2P manner, no DNS name resolution and stuff. Transparent "just works" type of communication.

Using the PIS one would mark certain types of information managed by the PIS as "public", and other PIS (those from your friends, co-workers, company , ...) would

query any "new" stuff and incorporate this new stuff into their own information store.

Using the Skype API, this could work out pretty nifty:

You start up your PIS instance
the PIS instance checks whether or not you're online with Skype
if you're online:
check if my buddies are online and have their PIS started
fetch the new stuff (contact info's, phone numbers, events, blogs, ...)

So, in theory, if you start up your PIS, it begins to sync with your peer PIS from your friends. That way, contacts and events stay synchronized, even if you're not online all the time, and even if you're behind a firewall.

You could also have more than one PIS, one on the road on your laptop, one at home, etc.

Nifty.

I think I'll play around with this stuff a bit. I've found no python bindings yet, but hey, at last we all have ctypes now with python 2.5, don't we ;)

Horrible week.

2006-10-21T15:29:00.000+02:00

Phew, this was a horrible week.

Really.

First, I had to boot our old SGI Octane machine because a customer wanted a new release of an CAD Model analysis tool we wrote. Thanks god it booted. And, thanks god again I didn't threw that pile of old, heavy, bulk and green hardware on the junkyard. Pure laziness, oh my.

Actually the recompile of our internal C based libs ran very smooth, and the python part of the program ran immediately. Testing took some time, though. That machine is no speed demon, thus analyzing a mere 24MiB worth of 3d CAD data took well over an hour. It produced tons of trace, using my hackery of sys.settrace() and my C tracing library. Well, I shipped it.

Then another customer filled their trac with tickets, all assigned to me. They're using a web application we wrote using Plone as an base to manage CAD drawings. They have lots of CAD drawings. Unfortunately they use Plone 2.1.1, which we migrated to from 2.0.5, which would well be worth a story of its own. Plone 2.1.1 seems to be a bad performer. We ran the profiling tools and killed the performance hogs, which turned out to have nothing to do with my application at all. Now they report that the command line tools I wrote to post/update/revoke drawings, which follow some sort of meta work flow on their own, does not delete drawings which became invalid. Oh well. Of course, on my test system the stuff works. Grrr.

On top of that I try to wrap my head around the new (?) and shiny (??) new python web application frameworks. Currently I'm investigating the possibilities of twisted, zope 3, turbogears and the new WSGI shipped with python 2.5. Much to learn. It seems I've missed most of the new development in that area which happened the last 1 1/2 years or so.

Google's GData API

2006-10-21T15:23:00.000+02:00

I'm currently investigating the use of the GData API when it comes to create a command line blog posting tool written in pure python.

Well, it seems that there's no Python API for the pure Google Data stuff yet (I know about the Google Base Data API).

But from what I've understood this is all pure HTML POST and GET using an extended version of the ATOM XML schema. Using urllib and ElementTree in python its just plain simple to actually get your blog feeds. They've even built in some sort of query language.

Unfortunately, to authenticate with Google using your Google account, there's a bit more involved. The Java guys have it better here: There's a nice high-level API. Doh.

Authentication is, as far as I understood, needed to post blog entries.

Note to myself:

Investigate Google Authentication wrt. Plone PAS
provide a pure python script which posts to a blog
hook script into a plone work flow to publish atricles from plone to blogs. Create a new content type for that, track blog post IDs we got back from google inside plone. Use Five and notifiers to update blog if Plone document changes. This would be a PIS module.

My own blog. I'm getting old and weird probably.

2006-10-17T02:09:00.000+02:00

Oh well. Just playing around. I've started a blog, oh my.

Well, I've read seome very good posts on blogs recently, specially when it comes
to my current interest in python web development.

For starters you might try effbot.org, a very extensive blog from Fredrik Lundh

Black.

2006-10-17T02:07:00.000+02:00

Yes, This is all black. I love black.

And courier. I love terminals.

Hmmm, email-to-blog?

2006-10-17T02:01:00.001+02:00

Nifty. But I'll have to remind myself to remove my signature from emails.