Posts about open source

Ubuntu One APIs by Example (part 1)

One of the nice things about working at Canonical is that we produce open source software. I, specifically, work in the team that does the desktop clients for Ubuntu One which is a really cool job, and a really cool piece of software. However, one thing not enough people know, is that we offer damn nice APIs for developers. We have to, since all our client code is open source, so we need those APIs for ourselves.

So, here is a small tutorial about using some of those APIs. I did it using Python and PyQt for several reasons:

  • Both are great tools for prototyping
  • Both have good support for the required stuff (DBus, HTTP, OAuth)
  • It's what I know and enjoy. Since I did this code on a sunday, I am not going to use other things.

Having said that, there is nothing python-specific or Qt-specific in the code. Where I do a HTTP request using QtNetwork, you are free to use libsoup, or whatever.

So, on to the nuts and bolts. The main pieces of Ubuntu One, from a infrastructure perspective, are Ubuntu SSO Client, that handles user registration and login, and SyncDaemon, which handles file synchronization.

To interact with them, on Linux, they offer DBus interfaces. So, for example, this is a fragment of code showing a way to get the Ubuntu One credentials (this would normally be part of an object's __init__):

# Get the session bus
bus = dbus.SessionBus()

:
:
:

# Get the credentials proxy and interface
self.creds_proxy = bus.get_object("com.ubuntuone.Credentials",
                        "/credentials",
                        follow_name_owner_changes=True)

# Connect to signals so you get a call when something
# credential-related happens
self.creds_iface = dbus.Interface(self.creds_proxy,
    "com.ubuntuone.CredentialsManagement")
self.creds_proxy.connect_to_signal('CredentialsFound',
    self.creds_found)
self.creds_proxy.connect_to_signal('CredentialsNotFound',
    self.creds_not_found)
self.creds_proxy.connect_to_signal('CredentialsError',
    self.creds_error)

# Call for credentials
self._credentials = None
self.get_credentials()

You may have noticed that get_credentials doesn't actually return the credentials. What it does is, it tells SyncDaemon to fetch the credentials, and then, when/if they are there, one of the signals will be emitted, and one of the connected methods will be called. This is nice, because it means you don't have to worry about your app blocking while SyncDaemon is doing all this.

But what's in those methods we used? Not much, really!

def get_credentials(self):
    # Do we have them already? If not, get'em
    if not self._credentials:
        self.creds_proxy.find_credentials()
    # Return what we've got, could be None
    return self._credentials

def creds_found(self, data):
    # Received credentials, save them.
    print "creds_found", data
    self._credentials = data
    # Don't worry about get_quota yet ;-)
    if not self._quota_info:
        self.get_quota()

def creds_not_found(self, data):
    # No credentials, remove old ones.
    print "creds_not_found", data
    self._credentials = None

def creds_error(self, data):
    # No credentials, remove old ones.
    print "creds_error", data
    self._credentials = None

So, basically, self._credentials will hold a set of credentials, or None. Congratulations, we are now logged into Ubuntu One, so to speak.

So, let's do something useful! How about asking for how much free space there is in the account? For that, we can't use the local APIs, we have to connect to the servers, who are, after all, the ones who decide if you are over quota or not.

Access is controlled via OAuth. So, to access the API, we need to sign our requests. Here is how it's done. It's not particularly enlightening, and I did not write it, I just use it:

def sign_uri(self, uri, parameters=None):
    # Without credentials, return unsigned URL
    if not self._credentials:
        return uri
    if isinstance(uri, unicode):
        uri = bytes(iri2uri(uri))
    print "uri:", uri
    method = "GET"
    credentials = self._credentials
    consumer = oauth.OAuthConsumer(credentials["consumer_key"],
                                   credentials["consumer_secret"])
    token = oauth.OAuthToken(credentials["token"],
                             credentials["token_secret"])
    if not parameters:
        _, _, _, _, query, _ = urlparse(uri)
        parameters = dict(cgi.parse_qsl(query))
    request = oauth.OAuthRequest.from_consumer_and_token(
                                        http_url=uri,
                                        http_method=method,
                                        parameters=parameters,
                                        oauth_consumer=consumer,
                                        token=token)
    sig_method = oauth.OAuthSignatureMethod_HMAC_SHA1()
    request.sign_request(sig_method, consumer, token)
    print "SIGNED:", repr(request.to_url())
    return request.to_url()

And how do we ask for the quota usage? By accessing the https://one.ubuntu.com/api/quota/ entry point with the proper authorization, we would get a JSON dictionary with total and used space. So, here's a simple way to do it:

    # This is on __init__
    self.nam = QtNetwork.QNetworkAccessManager(self,
        finished=self.reply_finished)

:
:
:

def get_quota(self):
    """Launch quota info request."""
    uri = self.sign_uri(QUOTA_API)
    url = QtCore.QUrl()
    url.setEncodedUrl(uri)
    self.nam.get(QtNetwork.QNetworkRequest(url))

Again, see how get_quota doesn't return the quota? What happens is that get_quota will launch a HTTP request to the Ubuntu One servers, which will, eventually, reply with the data. You don't want your app to block while you do that. So, QNetworkAccessManager will call self.reply_finished when it gets the response:

def reply_finished(self, reply):
    if unicode(reply.url().path()) == u'/api/quota/':
        # Handle quota responses
        self._quota_info = json.loads(unicode(reply.readAll()))
        print "Got quota: ", self._quota_info
        # Again, don't worry about update_menu yet ;-)
        self.update_menu()

What else would be nice to have? How about getting a call whenever the status of syncdaemon changes? For example, when sync is up to date, or when you get disconnected? Again, those are DBus signals we are connecting in our __init__:

self.status_proxy = bus.get_object(
    'com.ubuntuone.SyncDaemon', '/status')
self.status_iface = dbus.Interface(self.status_proxy,
    dbus_interface='com.ubuntuone.SyncDaemon.Status')
self.status_iface.connect_to_signal(
    'StatusChanged', self.status_changed)

# Get the status as of right now
self._last_status = self.process_status(
    self.status_proxy.current_status())

And what's status_changed?

def status_changed(self, status):
    print "New status:", status
    self._last_status = self.process_status(status)
    self.update_menu()

The process_status function is boring code to convert the info from syncdaemon's status into a human-readable thing like "Sync is up-to-date". So we store that in self._last_status and update the menu.

What menu? Well, a QSystemTrayIcon's context menu! What you have read are the main pieces you need to create something useful: a Ubuntu One tray app you can use in KDE, XFCE or openbox. Or, if you are on unity and install sni-qt, a Ubuntu One app indicator!

http://ubuntuone.com/7iXTbysoMM9PIUS9Ai4TNn

My Ubuntu One indicator in action.

You can find the source code for the whole example app at my u1-toys project in launchpad and here is the full source code (missing some icon resources, just get the repo)

Coming soon(ish), more example apps, and cool things to do with our APIs!

Creating a forum the easy way (32 LOC)

This is only the first part of a project to create the simplest (for me) software forum possible.

Here are the features I want:

  • Login using twitter / Facebook / Google / OpenID
  • Unlimited number of threads
  • Support for like / dislike both on threads and on posts
  • Avatars
  • HTML in comments
  • Mail the user on replies
  • RSS feeds for threads

You can see it in action at http://foro.netmanagers.com.ar (for a limited time only ;-)

And here is the code:

import bottle
import disqusapi as disqus
import json
shortname = 'magicmisteryforum'
api = disqus.DisqusAPI(open("key").read().strip())

@bottle.route('/', method='GET')
def index():
    msg = bottle.request.GET.get('msg', '')
    threads = api.forums.listThreads(forum=shortname, limit=100)
    print threads[0]
    return bottle.template('main.tpl', threads=threads, shortname=shortname, msg=msg)

@bottle.route('/new', method='POST')
def new():
    title = bottle.request.forms.get('title', None)
    if not title:
        bottle.redirect('/?msg=Missing%20Thread%20Name')
        return
    thread = api.threads.create(forum=shortname, title = title)
    thread_id = thread.__dict__['response']['id']
    # Redirecting to /thread/thread_id doesn't work
    # because threads take a few seconds to appear on the listing
    bottle.redirect('/')

@bottle.route('/thread/:id')
def thread(id):
    t = api.threads.details(thread=id)
    return bottle.template('thread.tpl', shortname=shortname, id=id, thread=t.__dict__['response'])

@bottle.route('/static/:path#.+#')
def server_static(path):
    return bottle.static_file(path, root='./static')

app = bottle.app()
app.catchall = False #Now most exceptions are re-raised within bottle.
bottle.run(host='184.82.108.14', port=80, app=app)

It requires Bottle and the Disqus python API

Of course, there is also a bit of templating involved, here is main.tpl and the thread.tpl. Since I suck at HTML, it uses Bluetrip CSS and it's more than simple enough to customize.

OF COURSE I AM CHEATING!

This thing is just a simple veneer around Disqus! More like a blog with comments and without posts than a forum! But ... what's missing to make this a real forum? It works, doesn't it? You could even use Disqus categories to create subforums...

All things considered, I think it's a cute hack.

And if you wait a few days, this will lead to something much more magical!

Full source code at http://magicforum.googlecode.com

OK, so THAT is how much browser I can put in 128 lines of code.

I have already posted a couple of times (1, 2) about De Vicenzo , an attempt to implement the rest of the browser, starting with PyQt's WebKit... limiting myself to 128 lines of code.

Of course I could do more, but I have my standards!

  • No using ;
  • No if whatever: f()

Other than that, I did a lot of dirty tricks, but right now, it's a fairly complete browser, and it has 127 lines of code (according to sloccount) so that's enough playing and it's time to go back to real work.

But first, let's consider how some features were implemented (I'll wrap the lines so they page stays reasonably narrow), and also look at the "normal" versions of the same (the "normal" code is not tested, please tell me if it's broken ;-).

This is not something you should learn how to do. In fact, this is almost a treatise on how not to do things. This is some of the least pythonic, less clear code you will see this week.

It is short, and it is expressive. But it is ugly.

I'll discuss this version.

Proxy Support

A browser is not much of a browser if you can't use it without a proxy, but luckily Qt's network stack has good proxy support. The trick was configuring it.

De Vicenzo supports HTTP and SOCKS proxies by parsing a http_proxy environment variable and setting Qt's application-wide proxy:

 proxy_url = QtCore.QUrl(os.environ.get('http_proxy', ''))
 QtNetwork.QNetworkProxy.setApplicationProxy(QtNetwork.QNetworkProxy(\
 QtNetwork.QNetworkProxy.HttpProxy if unicode(proxy_url.scheme()).startswith('http')\
 else QtNetwork.QNetworkProxy.Socks5Proxy, proxy_url.host(),\
 proxy_url.port(), proxy_url.userName(), proxy_url.password())) if\
'http_proxy' in os.environ else None

How would that look in normal code?

if 'http_proxy' in os.environ:
    proxy_url = QtCore.QUrl(os.environ['http_proxy'])
    if unicode(proxy_url.scheme()).starstswith('http'):
        protocol = QtNetwork.QNetworkProxy.HttpProxy
    else:
        protocol = QtNetwork.QNetworkProxy.Socks5Proxy
    QtNetwork.QNetworkProxy.setApplicationProxy(
        QtNetwork.QNetworkProxy(
            protocol,
            proxy_url.host(),
            proxy_url.port(),
            proxy_url.userName(),
            proxy_url.password()))

As you can see, the main abuses against python here are the use of the ternary operator as a one-line if (and nesting it), and line length.

Persistent Cookies

You really need this, since you want to stay logged into your sites between sessions. For this, first I needed to write some persistence mechanism, and then save/restore the cookies there.

Here's how the persistence is done (settings is a global QSettings instance):

def put(self, key, value):
    "Persist an object somewhere under a given key"
    settings.setValue(key, json.dumps(value))
    settings.sync()

def get(self, key, default=None):
    "Get the object stored under 'key' in persistent storage, or the default value"
    v = settings.value(key)
    return json.loads(unicode(v.toString())) if v.isValid() else default

It's not terribly weird code, except for the use of the ternary operator in the last line. The use of json ensures that as long as reasonable things are persisted, you will get them with the same type as you put them without needing to convert them or call special methods.

So, how do you save/restore the cookies? First, you need to access the cookie jar. I couldn't find whether there is a global one, or a per-webview one, so I created a QNetworkCookieJar in line 24 and assign it to each web page in line 107.

# Save the cookies, in the window's closeEvent
self.put("cookiejar", [str(c.toRawForm()) for c in self.cookies.allCookies()])

# Restore the cookies, in the window's __init__
self.cookies.setAllCookies([QtNetwork.QNetworkCookie.parseCookies(c)[0]\
for c in self.get("cookiejar", [])])

Here I confess I am guilty of using list comprehensions when a for loop would have been the correct thing.

I use the same trick when restoring the open tabs, with the added misfeature of using a list comprehension and throwing away the result:

# get("tabs") is a list of URLs
[self.addTab(QtCore.QUrl(u)) for u in self.get("tabs", [])]

Using Properties and Signals in Object Creation

This is a feature of recent PyQt versions: if you pass property names as keyword arguments when you create an object, they are assigned the value. If you pass a signal as a keyword argument, they are connected to the given value.

This is a really great feature that helps you create clear, local code, and it's a great thing to have. But if you are writing evil code... well, you can go to hell on a handbasket using it.

This is all over the place in De Vicenzo, and here's one example (yes, this is one line):

QtWebKit.QWebView.__init__(self, loadProgress=lambda v:\
(self.pbar.show(), self.pbar.setValue(v)) if self.amCurrent() else\
None, loadFinished=self.pbar.hide, loadStarted=lambda:\
self.pbar.show() if self.amCurrent() else None, titleChanged=lambda\
t: container.tabs.setTabText(container.tabs.indexOf(self), t) or\
(container.setWindowTitle(t) if self.amCurrent() else None))

Oh, boy, where do I start with this one.

There are lambda expressions used to define the callbacks in-place instead of just connecting to a real function or method.

There are lambdas that contain the ternary operator:

loadStarted=lambda:\
    self.pbar.show() if self.amCurrent() else None

There are lambdas that use or or a tuple to trick python into doing two things in a single lambda!

loadProgress=lambda v:\
(self.pbar.show(), self.pbar.setValue(v)) if self.amCurrent() else\
None

I won't even try to untangle this for educational purposes, but let's just say that line contains what should be replaced by 3 methods, and should be spread over 6 lines or more.

Download Manager

Ok, calling it a manager is overreaching, since you can't stop them once they start, but hey, it lets you download things and keep on browsing, and reports the progress!

First, on line 16 I created a bars dictionary for general bookkeeping of the downloads.

Then, I needed to delegate the unsupported content to the right method, and that's done in lines 108 and 109

What that does is basically that whenever you click on something WebKit can't handle, the method fetch will be called and passed the network request.

def fetch(self, reply):
    destination = QtGui.QFileDialog.getSaveFileName(self, \
        "Save File", os.path.expanduser(os.path.join('~',\
            unicode(reply.url().path()).split('/')[-1])))
    if destination:
        bar = QtGui.QProgressBar(format='%p% - ' +
            os.path.basename(unicode(destination)))
        self.statusBar().addPermanentWidget(bar)
        reply.downloadProgress.connect(self.progress)
        reply.finished.connect(self.finished)
        self.bars[unicode(reply.url().toString())] = [bar, reply,\
            unicode(destination)]

No real code golfing here, except for long lines, but once you break them reasonably, this is pretty much the obvious way to do it:

  • Ask for a filename
  • Create a progressbar, put it in the statusbar, and connect it to the download's progress signals.

Then, of course, we need ths progress slot, that updates the progressbar:

progress = lambda self, received, total:\
    self.bars[unicode(self.sender().url().toString())][0]\
    .setValue(100. * received / total)

Yes, I defined a method as a lambda to save 1 line. [facepalm]

And the finished slot for when the download is done:

def finished(self):
    reply = self.sender()
    url = unicode(reply.url().toString())
    bar, _, fname = self.bars[url]
    redirURL = unicode(reply.attribute(QtNetwork.QNetworkRequest.\
        RedirectionTargetAttribute).toString())
    del self.bars[url]
    bar.deleteLater()
    if redirURL and redirURL != url:
        return self.fetch(redirURL, fname)
    with open(fname, 'wb') as f:
        f.write(str(reply.readAll()))

Notice that it even handles redirections sanely! Beyond that, it just hides the progress bar, saves the data, end of story. The longest line is not even my fault!

There is a big inefficiency in that the whole file is kept in memory until the end. If you download a DVD image, that's gonna sting.

Also, using with saves a line and doesn't leak a file handle, compared to the alternatives.

Printing

Again Qt saved me, because doing this manually would have been a pain. However, it turns out that printing is just ... there? Qt, specially when used via PyQt is such an awesomely rich environment.

self.previewer = QtGui.QPrintPreviewDialog(\
    paintRequested=self.print_)
self.do_print = QtGui.QShortcut("Ctrl+p",\
    self, activated=self.previewer.exec_)

There's not even any need to golf here, that's exactly as much code as you need to hook Ctrl+p to make a QWebView print.

Other Tricks

There are no other tricks. All that's left is creating widgets, connecting things to one another, and enjoying the awesome experience of programming PyQt, where you can write a whole web browser (except the engine) in 127 lines of code.

De Vicenzo: A much cooler mini web browser.

It seems it was only a few days ago that I started this project. Oh, wait, yes, it was just a few days ago!

If you don't want to read that again, the idea is to see just how much code is needed to turn Qt's WebKit engine into a fully-fledged browser.

To do that, I set myself a completely arbitrary limit: 128 lines of code.

So, as of now, I declare it feature-complete.

The new features are:

  • Tabbed browsing (you can add/remove tabs)
  • Bookmarks (you can add/remove them, and choose them from a drop-down menu)

This is what already worked:

  • Zoom in (Ctrl++)
  • Zoom out (Ctrl+-)
  • Reset Zoom (Ctrl+=)
  • Find (Ctrl+F)
  • Hide find (Esc)
  • Buttons for back/forward and reload
  • URL entry that matches the page + autocomplete from history + smart entry (adds http://, that kind of thing)
  • Plugins support (including flash)
  • The window title shows the page title (without browser advertising ;-)
  • Progress bar for page loading
  • Statusbar that shows hovered links URL
  • Takes a URL on the command line, or opens http://python.org
  • Multiplatform (works in any place QtWebKit works)

So... how much code was needed for this? 87 LINES OF CODE

Or if you want the PEP8-compliant version, 115 LINES OF CODE.

Before anyone says it: yes, I know the rendering engine and the toolkit are huge. What I wrote is just the chrome around them, just like Arora, Rekonq, Galeon, Epiphany and a bunch of others do.

It's simple, minimalistic chrome, but it works pretty good, IMVHO.

Here it is in (buggy) action:

It's more or less feature-complete for what I expected to be achievable, but it still needs some fixes.

You can see the code at it's own home page: http://devicenzo.googlecode.com

How much web browser can you put in 128 lines of code?

UPDATE: If you read this and all you can say is "oh, he's just embedding WebKit", I have two things to tell you:

  1. Duh! Of course the 128 lines don't include the rendering engine, or the TCP implementation, or the GUI toolkit. This is about the rest of the browser, the part around the web rendering engine. You know, just like Arora, Rekonq, Epiphany, and everyone else that embeds webkit or mozilla does it? If you didn't get that before this explanation... facepalm.
  2. Get your favourite webkit fork and try to do this much with the same amount of code. I dare you! I double dog dare you!

Now back to the original article


Today, because of a IRC chat, I tried to find a 42-line web browser I had written a while ago. Sadly, the pastebin where I posted it was dead, so I learned a lesson: It's not a good idea to trust a pastebin as code repository

What I liked about that 42-line browser was that it was not the typical example, where someone dumps a Webkit view in a window, loads a page and tries to convince you he's cool. That one is only 7 lines of code:

import sys
from PyQt4 import QtGui,QtCore,QtWebKit
app=QtGui.QApplication(sys.argv)
wb=QtWebKit.QWebView()
wb.setUrl(QtCore.QUrl('http://www.python.org'))
wb.show()
sys.exit(app.exec_())

And if I wanted to make the code uglier, it could be done in 6.

But anyway, that 42-line browser actually looked useful!

This 42-line web browser, courtesy of #python and #qt -- http... on Twitpic

Those buttons you see actually worked correctly, enabling and disabling at the right moment, the URL entry changed when you clicked on links, and some other bits.

So, I have decided to start a small, intermittent project of code golf: put as much browser as I can in 128 lines of code (not counting comments or blanks), starting with PyQt4.

This has a useful purpose: I always suspected that if you assumed PyQt was part of the base system, most apps would fit in floppies again. This one fits on a 1.44MB floppy some 500 times (so you could use 360KB commodore floppies if you prefer!).

So far, I am at about 50 lines, and it has the following features:

  • Zoom in (Ctrl++)
  • Zoom out (Ctrl+-)
  • Reset Zoom (Ctrl+=)
  • Find (Ctrl+F)
  • Hide find (Esc)
  • Buttons for back/forward and reload
  • URL entry that matches the page + autocomplete from history + smart entry (adds http://, that kind of thing)
  • Plugins support (including flash)
  • The window title shows the page title (without browser advertising ;-)
  • Progress bar for page loading
  • Statusbar that shows hovered links URL
  • Takes a URL on the command line, or opens http://python.org
  • Multiplatform (works in any place QtWebKit works)

Missing are tabs and proxy support. I expect those will take another 40 lines or so, but I think it's probably the most featureful of these toy browsers.

The code... it's not all that hard. I am using lambda a lot, and I am using PyQt's keyword arguments for signal connection which makes lines long, but not hard. It could be made much smaller!

Here it is in action:

And here's the code:

#!/usr/bin/env python
"A web browser that will never exceed 128 lines of code. (not counting blanks)"

import sys
from PyQt4 import QtGui,QtCore,QtWebKit

class MainWindow(QtGui.QMainWindow):
    def __init__(self, url):
        QtGui.QMainWindow.__init__(self)
        self.sb=self.statusBar()

        self.pbar = QtGui.QProgressBar()
        self.pbar.setMaximumWidth(120)
        self.wb=QtWebKit.QWebView(loadProgress = self.pbar.setValue, loadFinished = self.pbar.hide, loadStarted = self.pbar.show, titleChanged = self.setWindowTitle)
        self.setCentralWidget(self.wb)

        self.tb=self.addToolBar("Main Toolbar")
        for a in (QtWebKit.QWebPage.Back, QtWebKit.QWebPage.Forward, QtWebKit.QWebPage.Reload):
            self.tb.addAction(self.wb.pageAction(a))

        self.url = QtGui.QLineEdit(returnPressed = lambda:self.wb.setUrl(QtCore.QUrl.fromUserInput(self.url.text())))
        self.tb.addWidget(self.url)

        self.wb.urlChanged.connect(lambda u: self.url.setText(u.toString()))
        self.wb.urlChanged.connect(lambda: self.url.setCompleter(QtGui.QCompleter(QtCore.QStringList([QtCore.QString(i.url().toString()) for i in self.wb.history().items()]), caseSensitivity = QtCore.Qt.CaseInsensitive)))

        self.wb.statusBarMessage.connect(self.sb.showMessage)
        self.wb.page().linkHovered.connect(lambda l: self.sb.showMessage(l, 3000))

        self.search = QtGui.QLineEdit(returnPressed = lambda: self.wb.findText(self.search.text()))
        self.search.hide()
        self.showSearch = QtGui.QShortcut("Ctrl+F", self, activated = lambda: (self.search.show() , self.search.setFocus()))
        self.hideSearch = QtGui.QShortcut("Esc", self, activated = lambda: (self.search.hide(), self.wb.setFocus()))

        self.quit = QtGui.QShortcut("Ctrl+Q", self, activated = self.close)
        self.zoomIn = QtGui.QShortcut("Ctrl++", self, activated = lambda: self.wb.setZoomFactor(self.wb.zoomFactor()+.2))
        self.zoomOut = QtGui.QShortcut("Ctrl+-", self, activated = lambda: self.wb.setZoomFactor(self.wb.zoomFactor()-.2))
        self.zoomOne = QtGui.QShortcut("Ctrl+=", self, activated = lambda: self.wb.setZoomFactor(1))
        self.wb.settings().setAttribute(QtWebKit.QWebSettings.PluginsEnabled, True)

        self.sb.addPermanentWidget(self.search)
        self.sb.addPermanentWidget(self.pbar)
        self.wb.load(url)


if __name__ == "__main__":
    app=QtGui.QApplication(sys.argv)
    if len(sys.argv) > 1:
        url = QtCore.QUrl.fromUserInput(sys.argv[1])
    else:
        url = QtCore.QUrl('http://www.python.org')
    wb=MainWindow(url)
    wb.show()
    sys.exit(app.exec_())

The Outhouse and the Mall

Wearing the software engineer's hat: Code is the most trivial and the least important part of a feature.

—Michael Iatrou

Michael tweeted that, I replied, he replied, but what the heck, I think sometimes things can be better explained in more than 140 characters, thus this post [1].

So, why the mall and the outhouse? Because when we talk about software and code and features, we are not all talking about the same thing. Imagine if I told you that bricks are trivial. After all, they have existed in their current form for thousands of years, they are pretty simple to manufacture, and have no interesting features, really, except a certain resistence.

Now, suppose you are building an outhouse. Since you are a funny guy, you want to build an actual brick outhouse so you will use bricks to do it.

Now, since bricks are so boring, you may feel compelled to believe bricks are the least important part of your edifice, and that the overall design is more important. Should you carve a moon-shaped hole in the door? How deep should the latrine be?

However, that position is fatally flawed, since if you ignore those trivial, boring bricks, all you have is shit in a hole in the ground. That is because you are considering the bricks as just a mean to your end. You only care about the bricks insofar as they help you realize your grand outhouse vision. I am here to tell you that you are wrong.

The first way in which you are wrong is in that artificial separation between means and ends. Everyone is familiar with the ethical conundrum about whether the ends justify the means, but that's garbage. That'swhat you say when you try to convince yourself that doing things haphazardly is ok, because what you do is just the means to whatever other thing is the end. Life is not so easily divided into things that matter and things that don't.

Your work, your creation is not just some ideal isolated end towards which you travel across a sea of dirty means, trying to keep your silver armour clean. It's one whole thing. You are creating the means, you are creating your goal, you are responsible for both, and if you use shoddy bricks, your outhouse should shame you.

In the same way, if you do crappy code, your feature is demeaned. It may even work, but you will know it's built out of crap. You will know you will have to fix and maintain that crap for years to come, or, if you are lucky, ruin your karma by dumping it on the head of some poor sucker who follows your steps.

I am pretty much a materialist. If you remove the code, you don't have a feature, or software, you have a concept, maybe an idea, perhaps a design (or maybe not) but certainly not software, just like you don't have a brick outhouse without piling some damn bricks one on top of the other.

I always say, when I see someone calling himself a software engineer, that I am merely a software carpenter. I know my tools, I care about them, I use them as well as I can according to my lights [2] and I try to produce as good a piece of furniture as I can with what I am given.

This tends to produce humble software, but it's software that has one redeeming feature: it knows what it should do, and does it as well as I can make it. For example, I wrote rst2pdf. It's a program that takes some sort of text, and produces PDF files. It does that as well as I could manage. It does nothing else. It works well or not, but it is what it is, it has a purpose, a description and a goal, and I have tried to achieve that goal without embarrasing myself.

My programs are outhouses, made of carefully selected and considered bricks. They are not fancy, but they are what they are and you know it just by looking at them. And if you ever need an outhouse, well, an outhouse is what you should get.

Also, people tend to do weird stuff with them I never expected, but that's just the luck of the analogy.

But why did I mention malls in the title? Because malls are not outhouses. Malls are not done with a goal by themselves beyond making money for its builders. The actual function of a piece of mall is not even known when it's being built. Will this be a McDonalds, or will it be a comic book store? Who knows!

A mall is built quickly with whatever makes sense moneywise, and it should look bland and recognisable, to not scare the herd. It's a building made for pedestrians, but it's intended to confuse them and make the path form A to B as long and meandering as possible. The premises on which its design is based are all askew, corrupted and self-contradicting.

They also give builders a chance to make lots of money. Or to lose lots of money.

Nowadays, we live in an age of mall software. People build startups, get financing, build crappy software and sometimes they hit it big (Twitter, Facebook) or, more likely, fade into obscurity leaving behind nothing at all, except money lost and sad programmers who spent nights coding stuff noone will ever see or use, and not much else.

Far from me saying startups are not a noble or worthy endeavour. They are! It's just that people who work on them should realize that they are not building software. That's why code doesn't look important to them, because they are actually selling eyeballs to advertisers, or collected personal data from their users to whoever buys that, or captive public for game developers, or whatever your business model says (if you have one!).

They are building malls, where the value is not in the building, which is pretty ghastly and useless by itself, but on the people in it, those who rent space in the mall, those who will use the mall, the software, the social network, whatever it is you are building.

Twitter is not software, Facebook is not software. If they were, identi.ca and diaspora would be bigger! What they are is people in one place, like a mall is not a real building, but a collection of people under a roof.

So, there is nothing wrong with building malls. Just remember that your ends and your means are one and a whole, that code is important, that without code Facebook and Twitter don't work, and that without people they are a badland, and know what you are doing.

Because the only hard thing in life is knowing what you want to do. The rest is the easy part. And because malls without toilets suck.

[1] If you really want to see the whole conversation, it's here: http://bettween.com/ralsina/iatrou (if anyone knows a better conversation tracker please post it in a comment).
[2] Yet here I am, an Engineering Manager at Canonical. Sorry guys!

A new stage, blah blah blah

You surely have seen a million posts like this one. Hacker X starts saying what a great time he had at company Y/college/mom's basement/the circus and how he will always miss the people there but anyway it's great to look forward to the challenge of life at company Z/unemployment/feng shui consulting/elefant excrement shoveling.

Well, this one is pretty much the same thing.

I started working at Canonical today. Yes, Canonical. The Ubuntu guys. You may wonder what a dyed-in-the-wool KDE guy is going to do there. Well, it's a job, dude, the 90s called and they want their flamefest back.

I am the new "Engineering Manager for the Desktop+ group". What the heck is that? Well, my job is to help a bunch of talented people I like (at least the ones I've known so far ;-) deliver cool software.

I will probably not be coding much, since this is a grownup job, the kind where instead of lines of code you are supposed to develop gastric ulcers and receding hairlines while you herd cats to the closest cat shed, but I will probably manage to do something, sometimes.

This position came at a good time for me. My kid is going to be 4 next year and go to school all day. What the heck was I gonna do at home all day then, watch anime? Build killer robots? Plan how to conquer the world?

And what happens to my previous job? Well... I still have it somehow. I own a piece of Net Managers (http://netmanagers.com.ar) but I will be stepping away from the daily management and operation of the business.

So, basically, I intend to take the money and dump the work on the backs of my capable partners (just kidding). In any case, the company can work just as well without me since we can now maybe hire an employee instead of paying me, so it's win/win ;-)

On other news, I will still work in the same table as the last 5 years, doing some of the same things, with different people. It doesn't sound so big when said like that, uh? Well, I will travel more, and there are interesting challenges in this new job.

In short: canonical, little coding, still own netmanagers, happy guy.

Charla: docutils / rst y sus amigos

Again, spanish only because it's a video... in spanish.

Resulta que me olvidé que sí habían grabado mi charla de docutils y compañia. Gracias a Germán por hacerme acordar y mostrarme adonde estaba!

Y ... acá está:

Rst2pdf 0.16 is out!

Finally, a new release of rst2pdf!

You can get it at its site: http://rst2pdf.googlecode.com

rst2pdf is a tool to convert restructured text (a light, cool markup language) to PDF using reportlab instead of LaTeX.

It has been used for many things, from books, to magazines, to brochures, to manuals, to websites and has lots of features:

  • Font embedding (TTF or Type1 fonts)
  • Cascading Stylesheets
  • Extremely flexible plugin architecture (you can do things like render the headings from arbitrary SVG files!)
  • Sphinx integration.
  • Configurable page layouts
  • Custom cover pages via templates
  • And much, much more...

The biggest change in 0.16 is surely the improved support for Sphinx 1.0.x so if you are using Sphinx, you really want this version.

Also, it has a ton of bugfixes, and a few minor but useful new features.

Here's the whole changelog if you don't believe me:

  • Fixed Issue 343: Plugged memory leak in the RSON parser.
  • Fix for Issue 287: there is still a corner case if you have two sections with the same title, at the same level, in the same page, in different files where the links will break.
  • Fixed Issue 367: german-localized dates are MM. DD. YYYY so when used in sphinx's template cover they appeared weird, like a list item. Fixed with a minor workaround in the template.
  • Fixed Issue 366: links to "#" make no sense on a PDF file
  • Made definitions from definition lists more stylable.
  • Moved definition lists to SplitTables, so you can have very long definitions.
  • Fixed Issue 318: Implemented Domain specific indexes for Sphinx 1.0.x
  • Fixed Index links when using Sphinx/pdfbuilder.
  • Fixed Issue 360: Set literal.wordWrap to None by default so it doesn't inherit wordWrap CJK when you use the otherwise correct japanese settings. In any case, literal blocks are not supposed to wrap at all.
  • Switched pdfbuilder to use SplitTables by default (it made no sense not to do it)
  • Fixed Issue 365: some TTF fonts don't validate but they work anyway.
  • Set a valid default baseurl for Sphinx (makes it much faster!)
  • New feature: --use-numbered-links to show section numbers in links to sections, like "See section 2.3 Termination"
  • Added stylesheets for landscape paper sizes (i.e: a4-landscape.style)
  • Fixed Issue 364: Some options not respected when passed in per-doc options in sphinx.
  • Fixed Issue 361: multiple linebreaks in line blocks were collapsed.
  • Fixed Issue 363: strange characters in some cases in math directive.
  • Fixed Issue 362: Smarter auto-enclosing of equations in $...$
  • Fixed Issue 358: --real--footnotes defaults to False, but help text indicates default is True
  • Fixed Issue 359: Wrong --fit-background-mode help string
  • Fixed Issue 356: missing cells if a cell spawns rows and columns.
  • Fixed Issue 349: Work correctly with languages that are available in form aa_bb and not aa (example: zh_cn)
  • Fixed Issue 345: give file/line info when there is an error in a raw PDF directive.
  • Fixed Issue 336: JPEG images should work even without PIL (but give a warning because sizes will probably be wrong)
  • Fixed Issue 351: footnote/citation references were generated incorrectly, which caused problems if there was a citation with the same text as a heading.
  • Fixed Issue 353: better handling of graphviz, so that it works without vectorpdf but gives a warning about it.
  • Fixed Issue 354: make todo_node from sphinx customizable.
  • Fixed bug where nested lists broke page layout if the page was small.
  • Smarter --inline-links option
  • New extension: fancytitles, see http://lateral.netmanagers.com.ar/weblog/posts/BB906.html
  • New feature: tab-width option in code-block directive (defaults to 8).
  • Fixed Issue 340: endnotes/footnotes were not styled.
  • Fixed Issue 339: class names using _ were not usable.
  • Fixed Issue 335: ugly crash when using images in some specific places (looks like a reportlab bug)
  • Fixed Issue 329: make the figure alignment/class attributes work more like LaTeX than HTML.
  • Fixed Issue 328: list item styles were being ignored.
  • Fixed Issue 186: new --use-floating-images makes images with :align: set work like in HTML, with the next flowable flowing beside it.
  • Fixed Issue 307: header/footer from stylesheet now supports inline rest markup and substitutions defined in the main document.
  • New pdf_toc_depth option for Sphinx/pdfbuilder
  • New pdf_use_toc option for Sphinx/pdfbuilder
  • Fixed Issue 308: compatibility with reportlab from SVN
  • Fixed Issue 323: errors in the config.sample made it work weird.
  • Fixed Issue 322: Image substitutions didn't work in document title.
  • Implemented Issue 321: underline and strikethrough available in stylesheet.
  • Fixed Issue 317: Ugly error message when file does not exist