Skip to main content

Ralsina.Me — Roberto Alsina's website

Posts about open source

Ubuntu One APIs by Example (part 1)

One of the nice things about work­ing at Canon­i­cal is that we pro­duce open source soft­ware. I, specif­i­cal­ly, work in the team that does the desk­top clients for Ubun­tu One which is a re­al­ly cool job, and a re­al­ly cool piece of soft­ware. How­ev­er, one thing not enough peo­ple know, is that we of­fer damn nice APIs for de­vel­op­er­s. We have to, since all our client code is open source, so we need those APIs for our­selves.

So, here is a small tu­to­ri­al about us­ing some of those APIs. I did it us­ing Python and PyQt for sev­er­al rea­son­s:

  • Both are great tools for pro­­to­­typ­ing

  • Both have good sup­­port for the re­quired stuff (D­Bus, HTTP, OAu­th)

  • It's what I know and en­joy. Since I did this code on a sun­­day, I am not go­ing to use oth­­er things.

Hav­ing said that, there is noth­ing python-spe­cif­ic or Qt-spe­cif­ic in the code. Where I do a HTTP re­quest us­ing Qt­Net­work, you are free to use lib­soup, or what­ev­er.

So, on to the nuts and bolt­s. The main pieces of Ubun­tu One, from a in­fra­struc­ture per­spec­tive, are Ubun­tu SSO Clien­t, that han­dles us­er reg­is­tra­tion and login, and Sync­Dae­mon, which han­dles file syn­chro­niza­tion.

To in­ter­act with them, on Lin­ux, they of­fer DBus in­ter­faces. So, for ex­am­ple, this is a frag­ment of code show­ing a way to get the Ubun­tu One cre­den­tials (this would nor­mal­ly be part of an ob­jec­t's __init__):

# Get the session bus
bus = dbus.SessionBus()

:
:
:

# Get the credentials proxy and interface
self.creds_proxy = bus.get_object("com.ubuntuone.Credentials",
                        "/credentials",
                        follow_name_owner_changes=True)

# Connect to signals so you get a call when something
# credential-related happens
self.creds_iface = dbus.Interface(self.creds_proxy,
    "com.ubuntuone.CredentialsManagement")
self.creds_proxy.connect_to_signal('CredentialsFound',
    self.creds_found)
self.creds_proxy.connect_to_signal('CredentialsNotFound',
    self.creds_not_found)
self.creds_proxy.connect_to_signal('CredentialsError',
    self.creds_error)

# Call for credentials
self._credentials = None
self.get_credentials()

You may have no­ticed that get_­cre­den­tials does­n't ac­tu­al­ly re­turn the cre­den­tial­s. What it does is, it tells Sync­Dae­mon to fetch the cre­den­tial­s, and then, when/if they are there, one of the sig­nals will be emit­ted, and one of the con­nect­ed meth­ods will be called. This is nice, be­cause it means you don't have to wor­ry about your app block­ing while Sync­Dae­mon is do­ing all this.

But what's in those meth­ods we used? Not much, re­al­ly!

def get_credentials(self):
    # Do we have them already? If not, get'em
    if not self._credentials:
        self.creds_proxy.find_credentials()
    # Return what we've got, could be None
    return self._credentials

def creds_found(self, data):
    # Received credentials, save them.
    print "creds_found", data
    self._credentials = data
    # Don't worry about get_quota yet ;-)
    if not self._quota_info:
        self.get_quota()

def creds_not_found(self, data):
    # No credentials, remove old ones.
    print "creds_not_found", data
    self._credentials = None

def creds_error(self, data):
    # No credentials, remove old ones.
    print "creds_error", data
    self._credentials = None

So, ba­si­cal­ly, self­._­cre­den­tials will hold a set of cre­den­tial­s, or None. Con­grat­u­la­tion­s, we are now logged in­to Ubun­tu One, so to speak.

So, let's do some­thing use­ful! How about ask­ing for how much free space there is in the ac­coun­t? For that, we can't use the lo­cal APIs, we have to con­nect to the server­s, who are, af­ter al­l, the ones who de­cide if you are over quo­ta or not.

Ac­cess is con­trolled via OAuth. So, to ac­cess the API, we need to sign our re­quest­s. Here is how it's done. It's not par­tic­u­lar­ly en­light­en­ing, and I did not write it, I just use it:

def sign_uri(self, uri, parameters=None):
    # Without credentials, return unsigned URL
    if not self._credentials:
        return uri
    if isinstance(uri, unicode):
        uri = bytes(iri2uri(uri))
    print "uri:", uri
    method = "GET"
    credentials = self._credentials
    consumer = oauth.OAuthConsumer(credentials["consumer_key"],
                                   credentials["consumer_secret"])
    token = oauth.OAuthToken(credentials["token"],
                             credentials["token_secret"])
    if not parameters:
        _, _, _, _, query, _ = urlparse(uri)
        parameters = dict(cgi.parse_qsl(query))
    request = oauth.OAuthRequest.from_consumer_and_token(
                                        http_url=uri,
                                        http_method=method,
                                        parameters=parameters,
                                        oauth_consumer=consumer,
                                        token=token)
    sig_method = oauth.OAuthSignatureMethod_HMAC_SHA1()
    request.sign_request(sig_method, consumer, token)
    print "SIGNED:", repr(request.to_url())
    return request.to_url()

And how do we ask for the quo­ta us­age? By ac­cess­ing the http­s://one.ubun­tu.­com/api/quo­ta/ en­try point with the prop­er au­tho­riza­tion, we would get a JSON dic­tio­nary with to­tal and used space. So, here's a sim­ple way to do it:

    # This is on __init__
    self.nam = QtNetwork.QNetworkAccessManager(self,
        finished=self.reply_finished)

:
:
:

def get_quota(self):
    """Launch quota info request."""
    uri = self.sign_uri(QUOTA_API)
    url = QtCore.QUrl()
    url.setEncodedUrl(uri)
    self.nam.get(QtNetwork.QNetworkRequest(url))

Again, see how get_quo­ta does­n't re­turn the quo­ta? What hap­pens is that get_quo­ta will launch a HTTP re­quest to the Ubun­tu One server­s, which will, even­tu­al­ly, re­ply with the da­ta. You don't want your app to block while you do that. So, QNet­workAc­cess­Man­ag­er will call self­.re­ply_fin­ished when it gets the re­spon­se:

def reply_finished(self, reply):
    if unicode(reply.url().path()) == u'/api/quota/':
        # Handle quota responses
        self._quota_info = json.loads(unicode(reply.readAll()))
        print "Got quota: ", self._quota_info
        # Again, don't worry about update_menu yet ;-)
        self.update_menu()

What else would be nice to have? How about get­ting a call when­ev­er the sta­tus of sync­dae­mon changes? For ex­am­ple, when sync is up to date, or when you get dis­con­nect­ed? Again, those are DBus sig­nals we are con­nect­ing in our __init__:

self.status_proxy = bus.get_object(
    'com.ubuntuone.SyncDaemon', '/status')
self.status_iface = dbus.Interface(self.status_proxy,
    dbus_interface='com.ubuntuone.SyncDaemon.Status')
self.status_iface.connect_to_signal(
    'StatusChanged', self.status_changed)

# Get the status as of right now
self._last_status = self.process_status(
    self.status_proxy.current_status())

And what's sta­tus_changed?

def status_changed(self, status):
    print "New status:", status
    self._last_status = self.process_status(status)
    self.update_menu()

The pro­cess_s­ta­tus func­tion is bor­ing code to con­vert the in­fo from sync­dae­mon's sta­tus in­to a hu­man-read­able thing like "Sync is up­-­to-­date". So we store that in self­._last_s­ta­tus and up­date the menu.

What menu? Well, a QSys­tem­Tray­I­con's con­text menu! What you have read are the main pieces you need to cre­ate some­thing use­ful: a Ubun­tu One tray app you can use in KDE, XFCE or open­box. Or, if you are on uni­ty and in­stall sni-qt, a Ubun­tu One app in­di­ca­tor!

http://ubuntuone.com/7iXTbysoMM9PIUS9Ai4TNn

My Ubun­tu One in­di­ca­tor in ac­tion.

You can find the source code for the whole ex­am­ple app at my u1-­toys project in launch­pad and here is the full source code (miss­ing some icon re­sources, just get the re­po)

Com­ing soon(ish), more ex­am­ple app­s, and cool things to do with our APIs!

Creating a forum the easy way (32 LOC)

This is on­ly the first part of a project to cre­ate the sim­plest (for me) soft­ware fo­rum pos­si­ble.

Here are the fea­tures I wan­t:

  • Lo­­gin us­ing twit­ter / Face­­book / Google / OpenID

  • Un­lim­it­ed num­ber of threads

  • Sup­­port for like / dis­­­like both on threads and on posts

  • Avatars

  • HTML in com­­ments

  • Mail the us­er on replies

  • RSS feeds for threads

You can see it in ac­tion at http://­foro.net­man­ager­s.­com.ar (for a lim­it­ed time on­ly ;-)

And here is the code:

import bottle
import disqusapi as disqus
import json
shortname = 'magicmisteryforum'
api = disqus.DisqusAPI(open("key").read().strip())

@bottle.route('/', method='GET')
def index():
    msg = bottle.request.GET.get('msg', '')
    threads = api.forums.listThreads(forum=shortname, limit=100)
    print threads[0]
    return bottle.template('main.tpl', threads=threads, shortname=shortname, msg=msg)

@bottle.route('/new', method='POST')
def new():
    title = bottle.request.forms.get('title', None)
    if not title:
        bottle.redirect('/?msg=Missing%20Thread%20Name')
        return
    thread = api.threads.create(forum=shortname, title = title)
    thread_id = thread.__dict__['response']['id']
    # Redirecting to /thread/thread_id doesn't work
    # because threads take a few seconds to appear on the listing
    bottle.redirect('/')

@bottle.route('/thread/:id')
def thread(id):
    t = api.threads.details(thread=id)
    return bottle.template('thread.tpl', shortname=shortname, id=id, thread=t.__dict__['response'])

@bottle.route('/static/:path#.+#')
def server_static(path):
    return bottle.static_file(path, root='./static')

app = bottle.app()
app.catchall = False #Now most exceptions are re-raised within bottle.
bottle.run(host='184.82.108.14', port=80, app=app)

It re­quires Bot­tle and the Dis­qus python API

Of course, there is al­so a bit of tem­plat­ing in­volved, here is main.t­pl and the thread­.t­pl. Since I suck at HTM­L, it us­es Bluetrip CSS and it's more than sim­ple enough to cus­tom­ize.

OF COURSE I AM CHEAT­ING!

This thing is just a sim­ple ve­neer around Dis­qus! More like a blog with com­ments and with­out posts than a fo­rum! But ... what's miss­ing to make this a re­al fo­rum? It work­s, does­n't it? You could even use Dis­qus cat­e­gories to cre­ate sub­fo­rum­s...

All things con­sid­ered, I think it's a cute hack.

And if you wait a few days, this will lead to some­thing much more mag­i­cal!

Full source code at http://­mag­ic­fo­rum.­google­code.­com

New golfing challenge: PatoCabrera

In the spir­it of the De Vi­cen­zo web browser, I am start­ing a new pro­gram, called Pa­to Cabr­era. Here are the rules:

  • Twit­ter client (no iden­ti.­­ca in the first ver­­sion, but to be added lat­er)

  • Has these fea­­tures: http://­­paste­bin.lug­­men.org.ar/6464

  • Has to be im­­ple­­men­t­ed be­­fore April 4th

  • Smal­l­­er than 16384 bytes (of python code) but may be larg­er be­­cause of art­­work.

Let's see how it works :-)

OK, so THAT is how much browser I can put in 128 lines of code.

I have al­ready post­ed a cou­ple of times (1, 2) about De Vi­cen­zo , an at­tempt to im­ple­ment the rest of the browser, start­ing with PyQt's We­bKit... lim­it­ing my­self to 128 lines of code.

Of course I could do more, but I have my stan­dard­s!

  • No us­ing ;

  • No if what­ev­er: f()

Oth­er than that, I did a lot of dirty trick­s, but right now, it's a fair­ly com­plete browser, and it has 127 lines of code (ac­cord­ing to sloc­coun­t) so that's enough play­ing and it's time to go back to re­al work.

But first, let's con­sid­er how some fea­tures were im­ple­ment­ed (I'll wrap the lines so they page stays rea­son­ably nar­row), and al­so look at the "nor­mal" ver­sions of the same (the "nor­mal" code is not test­ed, please tell me if it's bro­ken ;-).

This is not some­thing you should learn how to do. In fac­t, this is al­most a trea­tise on how not to do things. This is some of the least python­ic, less clear code you will see this week.

It is short, and it is ex­pres­sive. But it is ug­ly.

I'll dis­cuss this ver­sion.

Proxy Support

A brows­er is not much of a brows­er if you can't use it with­out a prox­y, but luck­i­ly Qt's net­work stack has good proxy sup­port. The trick was con­fig­ur­ing it.

De Vicenzo supports HTTP and SOCKS proxies by parsing a http_proxy environment variable and setting Qt's application-wide proxy:

 proxy_url = QtCore.QUrl(os.environ.get('http_proxy', ''))
 QtNetwork.QNetworkProxy.setApplicationProxy(QtNetwork.QNetworkProxy(\
 QtNetwork.QNetworkProxy.HttpProxy if unicode(proxy_url.scheme()).startswith('http')\
 else QtNetwork.QNetworkProxy.Socks5Proxy, proxy_url.host(),\
 proxy_url.port(), proxy_url.userName(), proxy_url.password())) if\
'http_proxy' in os.environ else None

How would that look in nor­mal code?

if 'http_proxy' in os.environ:
    proxy_url = QtCore.QUrl(os.environ['http_proxy'])
    if unicode(proxy_url.scheme()).starstswith('http'):
        protocol = QtNetwork.QNetworkProxy.HttpProxy
    else:
        protocol = QtNetwork.QNetworkProxy.Socks5Proxy
    QtNetwork.QNetworkProxy.setApplicationProxy(
        QtNetwork.QNetworkProxy(
            protocol,
            proxy_url.host(),
            proxy_url.port(),
            proxy_url.userName(),
            proxy_url.password()))

As you can see, the main abus­es against python here are the use of the ternary op­er­a­tor as a one-­line if (and nest­ing it), and line length.

Persistent Cookies

You re­al­ly need this, since you want to stay logged in­to your sites be­tween ses­sion­s. For this, first I need­ed to write some per­sis­tence mech­a­nis­m, and then save/re­store the cook­ies there.

Here's how the persistence is done (settings is a global QSettings instance):

def put(self, key, value):
    "Persist an object somewhere under a given key"
    settings.setValue(key, json.dumps(value))
    settings.sync()

def get(self, key, default=None):
    "Get the object stored under 'key' in persistent storage, or the default value"
    v = settings.value(key)
    return json.loads(unicode(v.toString())) if v.isValid() else default

It's not terribly weird code, except for the use of the ternary operator in the last line. The use of json ensures that as long as reasonable things are persisted, you will get them with the same type as you put them without needing to convert them or call special methods.

So, how do you save/restore the cookies? First, you need to access the cookie jar. I couldn't find whether there is a global one, or a per-webview one, so I created a QNetworkCookieJar in line 24 and assign it to each web page in line 107.

# Save the cookies, in the window's closeEvent
self.put("cookiejar", [str(c.toRawForm()) for c in self.cookies.allCookies()])

# Restore the cookies, in the window's __init__
self.cookies.setAllCookies([QtNetwork.QNetworkCookie.parseCookies(c)[0]\
for c in self.get("cookiejar", [])])

Here I con­fess I am guilty of us­ing list com­pre­hen­sions when a for loop would have been the cor­rect thing.

I use the same trick when restor­ing the open tab­s, with the added mis­fea­ture of us­ing a list com­pre­hen­sion and throw­ing away the re­sult:

# get("tabs") is a list of URLs
[self.addTab(QtCore.QUrl(u)) for u in self.get("tabs", [])]

Using Properties and Signals in Object Creation

This is a fea­ture of re­cent PyQt ver­sion­s: if you pass prop­er­ty names as key­word ar­gu­ments when you cre­ate an ob­jec­t, they are as­signed the val­ue. If you pass a sig­nal as a key­word ar­gu­men­t, they are con­nect­ed to the giv­en val­ue.

This is a re­al­ly great fea­ture that helps you cre­ate clear, lo­cal code, and it's a great thing to have. But if you are writ­ing evil code... well, you can go to hell on a hand­bas­ket us­ing it.

This is all over the place in De Vi­cen­zo, and here's one ex­am­ple (yes, this is one line):

QtWebKit.QWebView.__init__(self, loadProgress=lambda v:\
(self.pbar.show(), self.pbar.setValue(v)) if self.amCurrent() else\
None, loadFinished=self.pbar.hide, loadStarted=lambda:\
self.pbar.show() if self.amCurrent() else None, titleChanged=lambda\
t: container.tabs.setTabText(container.tabs.indexOf(self), t) or\
(container.setWindowTitle(t) if self.amCurrent() else None))

Oh, boy, where do I start with this one.

There are lambda expressions used to define the callbacks in-place instead of just connecting to a real function or method.

There are lamb­das that con­tain the ternary op­er­a­tor:

loadStarted=lambda:\
    self.pbar.show() if self.amCurrent() else None

There are lambdas that use or or a tuple to trick python into doing two things in a single lambda!

loadProgress=lambda v:\
(self.pbar.show(), self.pbar.setValue(v)) if self.amCurrent() else\
None

I won't even try to un­tan­gle this for ed­u­ca­tion­al pur­pos­es, but let's just say that line con­tains what should be re­placed by 3 meth­od­s, and should be spread over 6 lines or more.

Download Manager

Ok, call­ing it a man­ag­er is over­reach­ing, since you can't stop them once they start, but hey, it lets you down­load things and keep on brows­ing, and re­ports the pro­gress!

First, on line 16 I created a bars dictionary for general bookkeeping of the downloads.

Then, I need­ed to del­e­gate the un­sup­port­ed con­tent to the right method, and that's done in lines 108 and 109

What that does is basically that whenever you click on something WebKit can't handle, the method fetch will be called and passed the network request.

def fetch(self, reply):
    destination = QtGui.QFileDialog.getSaveFileName(self, \
        "Save File", os.path.expanduser(os.path.join('~',\
            unicode(reply.url().path()).split('/')[-1])))
    if destination:
        bar = QtGui.QProgressBar(format='%p% - ' +
            os.path.basename(unicode(destination)))
        self.statusBar().addPermanentWidget(bar)
        reply.downloadProgress.connect(self.progress)
        reply.finished.connect(self.finished)
        self.bars[unicode(reply.url().toString())] = [bar, reply,\
            unicode(destination)]

No re­al code golf­ing here, ex­cept for long lines, but once you break them rea­son­ably, this is pret­ty much the ob­vi­ous way to do it:

  • Ask for a file­­name

  • Cre­ate a pro­­gress­bar, put it in the sta­­tus­bar, and con­nect it to the down­load­­'s progress sig­­nal­s.

Then, of course, we need ths progress slot, that updates the progressbar:

progress = lambda self, received, total:\
    self.bars[unicode(self.sender().url().toString())][0]\
    .setValue(100. * received / total)

Yes, I de­fined a method as a lamb­da to save 1 line. [facepalm]

And the finished slot for when the download is done:

def finished(self):
    reply = self.sender()
    url = unicode(reply.url().toString())
    bar, _, fname = self.bars[url]
    redirURL = unicode(reply.attribute(QtNetwork.QNetworkRequest.\
        RedirectionTargetAttribute).toString())
    del self.bars[url]
    bar.deleteLater()
    if redirURL and redirURL != url:
        return self.fetch(redirURL, fname)
    with open(fname, 'wb') as f:
        f.write(str(reply.readAll()))

No­tice that it even han­dles redi­rec­tions sane­ly! Be­yond that, it just hides the progress bar, saves the data, end of sto­ry. The long­est line is not even my fault!

There is a big in­ef­fi­cien­cy in that the whole file is kept in mem­o­ry un­til the end. If you down­load a DVD im­age, that's gonna sting.

Also, using with saves a line and doesn't leak a file handle, compared to the alternatives.

Printing

Again Qt saved me, be­cause do­ing this man­u­al­ly would have been a pain. How­ev­er, it turns out that print­ing is just ... there? Qt, spe­cial­ly when used via PyQt is such an awe­some­ly rich en­vi­ron­men­t.

self.previewer = QtGui.QPrintPreviewDialog(\
    paintRequested=self.print_)
self.do_print = QtGui.QShortcut("Ctrl+p",\
    self, activated=self.previewer.exec_)

There's not even any need to golf here, that's exactly as much code as you need to hook Ctrl+p to make a QWebView print.

Other Tricks

There are no oth­er trick­s. All that's left is cre­at­ing wid­get­s, con­nect­ing things to one an­oth­er, and en­joy­ing the awe­some ex­pe­ri­ence of pro­gram­ming PyQt, where you can write a whole web brows­er (ex­cept the en­gine) in 127 lines of code.

De Vicenzo: A much cooler mini web browser.

It seems it was on­ly a few days ago that I start­ed this projec­t. Oh, wait, yes, it was just a few days ago!

If you don't want to read that again, the idea is to see just how much code is need­ed to turn Qt's We­bKit en­gine in­to a ful­ly-fledged brows­er.

To do that, I set my­self a com­plete­ly ar­bi­trary lim­it: 128 lines of code.

So, as of now, I de­clare it fea­ture-­com­plete.

The new fea­tures are:

  • Tabbed brows­ing (y­ou can ad­d/re­­move tab­s)

  • Book­­marks (y­ou can ad­d/re­­move them, and choose them from a drop-­­down menu)

This is what al­ready worked:

  • Zoom in (C­tr­l++)

  • Zoom out (C­tr­l+-)

  • Re­set Zoom (C­tr­l+=)

  • Find (C­tr­l+F)

  • Hide find (Esc)

  • But­­tons for back­­/­­for­ward and reload

  • URL en­try that match­es the page + au­­to­­com­­plete from his­­to­ry + smart en­try (adds http://, that kind of thing)

  • Plug­ins sup­­port (in­­clud­ing flash)

  • The win­­dow ti­­tle shows the page ti­­tle (with­­out brows­er ad­ver­tis­ing ;-)

  • Progress bar for page load­­ing

  • Sta­­tus­bar that shows hov­­ered links URL

  • Takes a URL on the com­­mand line, or opens http://python.org

  • Mul­ti­­plat­­form (works in any place QtWe­bKit work­s)

So... how much code was need­ed for this? 87 LINES OF CODE

Or if you want the PEP8-­com­pli­ant ver­sion, 115 LINES OF CODE.

Be­fore any­one says it: yes, I know the ren­der­ing en­gine and the tool­kit are huge. What I wrote is just the chrome around them, just like Aro­ra, Rekon­q, Ga­le­on, Epiphany and a bunch of oth­ers do.

It's sim­ple, min­i­mal­is­tic chrome, but it works pret­ty good, IMVHO.

Here it is in (bug­gy) ac­tion:

It's more or less fea­ture-­com­plete for what I ex­pect­ed to be achiev­able, but it still needs some fix­es.

You can see the code at it's own home page: http://de­vi­cen­zo.­google­code.­com


Contents © 2000-2020 Roberto Alsina