Posts about python (old posts, page 36)

2012-03-13 02:17

Ubuntu One APIs by Example (part 1)

One of the nice things about working at Canonical is that we produce open source software. I, specifically, work in the team that does the desktop clients for Ubuntu One which is a really cool job, and a really cool piece of software. However, one thing not enough people know, is that we offer damn nice APIs for developers. We have to, since all our client code is open source, so we need those APIs for ourselves.

So, here is a small tutorial about using some of those APIs. I did it using Python and PyQt for several reasons:

  • Both are great tools for prototyping
  • Both have good support for the required stuff (DBus, HTTP, OAuth)
  • It's what I know and enjoy. Since I did this code on a sunday, I am not going to use other things.

Having said that, there is nothing python-specific or Qt-specific in the code. Where I do a HTTP request using QtNetwork, you are free to use libsoup, or whatever.

So, on to the nuts and bolts. The main pieces of Ubuntu One, from a infrastructure perspective, are Ubuntu SSO Client, that handles user registration and login, and SyncDaemon, which handles file synchronization.

To interact with them, on Linux, they offer DBus interfaces. So, for example, this is a fragment of code showing a way to get the Ubuntu One credentials (this would normally be part of an object's __init__):

# Get the session bus
bus = dbus.SessionBus()

:
:
:

# Get the credentials proxy and interface
self.creds_proxy = bus.get_object("com.ubuntuone.Credentials",
                        "/credentials",
                        follow_name_owner_changes=True)

# Connect to signals so you get a call when something
# credential-related happens
self.creds_iface = dbus.Interface(self.creds_proxy,
    "com.ubuntuone.CredentialsManagement")
self.creds_proxy.connect_to_signal('CredentialsFound',
    self.creds_found)
self.creds_proxy.connect_to_signal('CredentialsNotFound',
    self.creds_not_found)
self.creds_proxy.connect_to_signal('CredentialsError',
    self.creds_error)

# Call for credentials
self._credentials = None
self.get_credentials()

You may have noticed that get_credentials doesn't actually return the credentials. What it does is, it tells SyncDaemon to fetch the credentials, and then, when/if they are there, one of the signals will be emitted, and one of the connected methods will be called. This is nice, because it means you don't have to worry about your app blocking while SyncDaemon is doing all this.

But what's in those methods we used? Not much, really!

def get_credentials(self):
    # Do we have them already? If not, get'em
    if not self._credentials:
        self.creds_proxy.find_credentials()
    # Return what we've got, could be None
    return self._credentials

def creds_found(self, data):
    # Received credentials, save them.
    print "creds_found", data
    self._credentials = data
    # Don't worry about get_quota yet ;-)
    if not self._quota_info:
        self.get_quota()

def creds_not_found(self, data):
    # No credentials, remove old ones.
    print "creds_not_found", data
    self._credentials = None

def creds_error(self, data):
    # No credentials, remove old ones.
    print "creds_error", data
    self._credentials = None

So, basically, self._credentials will hold a set of credentials, or None. Congratulations, we are now logged into Ubuntu One, so to speak.

So, let's do something useful! How about asking for how much free space there is in the account? For that, we can't use the local APIs, we have to connect to the servers, who are, after all, the ones who decide if you are over quota or not.

Access is controlled via OAuth. So, to access the API, we need to sign our requests. Here is how it's done. It's not particularly enlightening, and I did not write it, I just use it:

def sign_uri(self, uri, parameters=None):
    # Without credentials, return unsigned URL
    if not self._credentials:
        return uri
    if isinstance(uri, unicode):
        uri = bytes(iri2uri(uri))
    print "uri:", uri
    method = "GET"
    credentials = self._credentials
    consumer = oauth.OAuthConsumer(credentials["consumer_key"],
                                   credentials["consumer_secret"])
    token = oauth.OAuthToken(credentials["token"],
                             credentials["token_secret"])
    if not parameters:
        _, _, _, _, query, _ = urlparse(uri)
        parameters = dict(cgi.parse_qsl(query))
    request = oauth.OAuthRequest.from_consumer_and_token(
                                        http_url=uri,
                                        http_method=method,
                                        parameters=parameters,
                                        oauth_consumer=consumer,
                                        token=token)
    sig_method = oauth.OAuthSignatureMethod_HMAC_SHA1()
    request.sign_request(sig_method, consumer, token)
    print "SIGNED:", repr(request.to_url())
    return request.to_url()

And how do we ask for the quota usage? By accessing the https://one.ubuntu.com/api/quota/ entry point with the proper authorization, we would get a JSON dictionary with total and used space. So, here's a simple way to do it:

    # This is on __init__
    self.nam = QtNetwork.QNetworkAccessManager(self,
        finished=self.reply_finished)

:
:
:

def get_quota(self):
    """Launch quota info request."""
    uri = self.sign_uri(QUOTA_API)
    url = QtCore.QUrl()
    url.setEncodedUrl(uri)
    self.nam.get(QtNetwork.QNetworkRequest(url))

Again, see how get_quota doesn't return the quota? What happens is that get_quota will launch a HTTP request to the Ubuntu One servers, which will, eventually, reply with the data. You don't want your app to block while you do that. So, QNetworkAccessManager will call self.reply_finished when it gets the response:

def reply_finished(self, reply):
    if unicode(reply.url().path()) == u'/api/quota/':
        # Handle quota responses
        self._quota_info = json.loads(unicode(reply.readAll()))
        print "Got quota: ", self._quota_info
        # Again, don't worry about update_menu yet ;-)
        self.update_menu()

What else would be nice to have? How about getting a call whenever the status of syncdaemon changes? For example, when sync is up to date, or when you get disconnected? Again, those are DBus signals we are connecting in our __init__:

self.status_proxy = bus.get_object(
    'com.ubuntuone.SyncDaemon', '/status')
self.status_iface = dbus.Interface(self.status_proxy,
    dbus_interface='com.ubuntuone.SyncDaemon.Status')
self.status_iface.connect_to_signal(
    'StatusChanged', self.status_changed)

# Get the status as of right now
self._last_status = self.process_status(
    self.status_proxy.current_status())

And what's status_changed?

def status_changed(self, status):
    print "New status:", status
    self._last_status = self.process_status(status)
    self.update_menu()

The process_status function is boring code to convert the info from syncdaemon's status into a human-readable thing like "Sync is up-to-date". So we store that in self._last_status and update the menu.

What menu? Well, a QSystemTrayIcon's context menu! What you have read are the main pieces you need to create something useful: a Ubuntu One tray app you can use in KDE, XFCE or openbox. Or, if you are on unity and install sni-qt, a Ubuntu One app indicator!

http://ubuntuone.com/7iXTbysoMM9PIUS9Ai4TNn

My Ubuntu One indicator in action.

You can find the source code for the whole example app at my u1-toys project in launchpad and here is the full source code (missing some icon resources, just get the repo)

Coming soon(ish), more example apps, and cool things to do with our APIs!

2012-03-06 02:40

So, rst2pdf is now 0.91

Turns out there was a major, show stopper bug in rst2pdf 0.90: sphinx support was absolutely broken. Not broken as in buggy, broken as in it had syntax errors.

So, 0.91 is now released, and only broken in the traditional sense. Enjoy!

2012-03-04 17:40

rst2pdf 0.90 is out

Yes, after many moons, it's out. Here is the (as usual) incomplete changelog:

  • Added raw HTML support, by Dimitri Christodoulou
  • Fixed Issue 422: Having no .afm files made font lookup slow.
  • Fixed Issue 411: Sometimes the windows registry has the font's abspath.
  • Fixed Issue 430: Using --config option caused other options to be ignored (by charles at cstanhope dot com)
  • Fixed Issue 436: Add pdf_style_path to sphinx (by [email protected])
  • Fixed Issue 428: page numbers logged as errors
  • Added support for many pygments options in code-block (by Joaquin Sorianello)
  • Implemented Issue 404: plantuml support
  • Issue 399: support sphinx's template path option
  • Fixed Issue 406: calls to the wrong logging function
  • Implemented Issue 391: New --section-header-depth option.
  • Fixed Issue 390: the --config option was ignored.
  • Added support for many pygments options in code-block (by Joaquin Sorianello)
  • Fixed Issue 379: Wrong style applied to paragraphs in definitions.
  • Fixed Issue 378: Multiline :address: were shown collapsed.
  • Implemented Issue 11: FrameBreak (and conditional FrameBreak)
  • The description of frames in page templates was just wrong.
  • Fixed Issue 374: in some cases, literal blocks were split inside a page, or the pagebreak came too early.
  • Fixed Issue 370: warning about sphinx.addnodes.highlightlang not being handled removed.
  • Fixed Issue 369: crash in hyphenator when specifying "en" as a language.
  • Compatibility fix to Sphinx 0.6.x (For python 2.7 docs)

This release did not focus on Sphinx bugs, so those are probably still there. Hopefully the next round is attacking those.

2012-02-24 02:38

PyQt Quickie: command line parsing

So, you are writing a PyQt app, and you want it to support command line arguments. So you do something like this:

opt_parser = OptionParser()
opt_parser.add_option("-q", dest="quickly", action="store_true",
    help="Do it quickly (default=False)")
(options, args) = opt_parser.parse_args(sys.argv)
app = QApplication(sys.argv)
:
:
:

Or maybe even QApplication([]). Ok, you are doing it wrong. And this is wrong in most tutorials, too. Why? Because Qt (and thus PyQt) supports a bunch of useful command line options already. So if you do it like in the first listing, and pass "-style=oxygen" or whatever, one of the following will happen.

  1. OptParser is going to tell you it's not a valid option and abort
  2. You will ignore the option and not do anything useful with it
  3. You will have your own -style option and do two things with it

All three outcomes are less than ideal.

The right way to do this is:

opt_parser = OptionParser()
opt_parser.add_option("-q", dest="quickly", action="store_true",
    help="Do it quickly (default=False)")
app = QApplication(sys.argv)
(options, args) = opt_parser.parse_args(app.arguments())
:
:
:

This way, you give PyQt a chance to process the options it recognizes, and, then, you get to handle the rest, because app.arguments() has all Qt options removed.

The bad side of this is, you will make --help slightly slower, since it will have to build a QApplication to do nothing, and you will have undocumented options. Solution for both problems left as an exercise.

2012-02-17 20:34

Scraping doesn't hurt

I am in general allergic to HTML, specially when it comes to parsing it. However, every now and then something comes up and it's fun to keep the muscles stretched.

So, consider the Ted Talks site. They have a really nice table with information about their talks, just in case you want to do something with them.

But how do you get that information? By scraping it. And what's an easy way to do it? By using Python and BeautifulSoup:

from BeautifulSoup import BeautifulSoup
import urllib

# Read the whole page.
data = urllib.urlopen('http://www.ted.com/talks/quick-list').read()
# Parse it
soup = BeautifulSoup(data)

# Find the table with the data
table = soup.findAll('table', attrs= {"class": "downloads notranslate"})[0]
# Get the rows, skip the first one
rows = table.findAll('tr')[1:]

items = []
# For each row, get the data
# And store it somewhere
for row in rows:
    cells = row.findAll('td')
    item = {}
    item['date'] = cells[0].text
    item['event'] = cells[1].text
    item['title'] = cells[2].text
    item['duration'] = cells[3].text
    item['links'] = [a['href'] for a in cells[4].findAll('a')]
    items.append(item)

And that's it! Surprisingly pain-free!

2012-02-17 03:18

To write, and to write what.

Some of you may know I have written about 30% of a book, called "Python No Muerde", available at http://nomuerde.netmanagers.com.ar (in spanish only).That book has stagnated for a long time.

On the other hand, I wrote a very popular series of posts, called PyQt by Example, which has (you guessed it) stagnated for a long time.

The main problem with the book was that I tried to cover way too much ground. When complete, it would be a 500 page book, and that would involve writing half a dozen example apps, some of them in areas I am no expert.

The main problem with the post series is that the example is lame (a TODO app!) and expanding it is boring.

¡So, what better way to fix both things at once, than to merge them!

I will leave Python No Muerde as it is, and will do a new book, called PyQt No Muerde. It will keep the tone and language of Python No Muerde, and will even share some chapters, but will focus on developing a PyQt app or two, instead of the much more ambitious goals of Python No Muerde. It will be about 200 pages.

I have acquired permission from my superiors (my wife) to work on this project a couple of hours a day, in the early morning. So, it may move forward, or it may not. This is, as usual, an experiment, not a promise.

2012-02-10 22:57

PyQt Quickie: Don't Get Garbage Collected

There is one area where Qt and Python (and in consequence PyQt) have major disagreements. That area is memory management.

While Qt has its own mechanisms to handle object allocation and disposal (the hierarchical QObject trees, smart pointers, etc.), PyQt runs on Python, so it has garbage collection.

Let's consider a simple example:

from PyQt4 import QtCore

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

If you run this, this is what will happen:

QProcess: Destroyed while process is still running.
The process is done!

Plus, the script never ends. Fun! The problem is that proc is being deleted at the end of launch_process because there are no more references to it.

Here is a better way to do it:

from PyQt4 import QtCore

processes = set([])

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

Here, we add a global processes set and add proc there so we always keep a reference to it. Now, the program works as intended. However, it still has an issue: we are leaking QProcess objects.

While in this case the leak is very short-lived, since we are ending the program right after the process ends, in a real program this is not a good idea.

So, we would need to add a way to remove proc from processes in finished. This is not as easy as it may seem. Here is an idea that will not work as you expect:

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # Remove the process from the global set when done
    proc.finished.connect(lambda: processes.remove(proc))
    # After it finishes, call finished
    proc.finished.connect(finished)

In this version, we will still leak proc, even though processes is empty! Why? Because we are keeping a reference to proc in the lambda!

I don't really have a good answer for that that doesn't involve turning everything into members of a QObject and using sender to figure out what process is ending, or using QSignalMapper. That version is left as an exercise.

2012-01-31 18:08

Garbage Collection Has Side Effects

Just a quick followup to The problem is is, is it not? This is not mine, I got it from reddit

This should really not surprise you:

>>> a = [1,2]
>>> b = [3,4]
>>> a is b
False
>>> a == b
False
>>> id(a) == id(b)
False

After all, a and b are completely different things. However:

>>> [1,2] is [3,4]
False
>>> [1,2] == [3,4]
False
>>> id([1,2]) == id([3,4])
True

Turns out that using literals, one of those things is not like the others.

First, the explanation so you understand why this happens. When you don't have any more references to a piece of data, it will get garbage collected, the memory will be freed, so it can be reused for other things.

In the first case, I am keeping references to both lists in the variables a and b. That means the lists have to exist at all times, since I can always say print a and python has to know what's in it.

In the second case, I am using literals, which means there is no reference to the lists after they are used. When python evaluates id([1,2]) == id([3,4]) it first evaluates the left side of the ==. After that is done, there is no need to keep [1,2] available, so it's deleted. Then, when evaluating the right side, it creates [3,4].

By pure chance, it will use the exact same place for it as it was using for [1,2]. So id will return the same value. This is just to remind you of a couple of things:

  1. a is b is usually (but not always) the same as id(a) == id(b)
  2. garbage collection can cause side effects you may not be expecting

2012-01-28 18:14

The problem is is. Is it not?

This has been a repeated discussion in the Python Argentina mailing list. Since it has not come up in a while, why not recap it, so the next time it happens people can just link here.

Some people for some reason do this:

>>> a = 2
>>> b = 2
>>> a == b
True
>>> a is b
True

And then, when they do this, they are surprised:

>>> a = 1000
>>> b = 1000
>>> a == b
True
>>> a is b
False

They are surprised because "2 is 2" makes more intuitive sense than "1000 is not 1000". This could be attributed to an inclination towards platonism, but really, it's because they don't know what is is.

The is operator is (on CPython) simply a memory address comparison. if objects a and b are the same exact chunk of memory, then they "are" each other. Since python pre-creates a bunch of small integers, then every 2 you create is really not a new 2, but the same 2 of last time.

This works because of two things:

  1. Integers are read-only objects. You can have as many variables "holding" the same 2, because they can't break it.
  2. In python, assignment is just aliasing. You are not making a copy of 2 when you do a = 2, you are just saying "a is another name for this 2 here".

This is surprising for people coming from other languages, like, say, C or C++. In those languages, a variable int a will never use the same memory space as another variable int b because a and b are names for specific bytes of memory, and you can change the contents of those bytes. On C and C++, integers are a mutable type. This 2 is not that 2, unless you do it intentionally using pointers.

In fact, the way assignment works on Python also leads to other surprises, more interesting in real life. For example, look at this session:

>>> def f(s=""):
...     s+='x'
...     return s
...
>>> f()
'x'
>>> f()
'x'
>>> f()
'x'

That is really not surprising. Now, let's make a very small change:

>>> def f(l=[]):
...     l.append('x')
...     return l
...
>>> f()
['x']
>>> f()
['x', 'x']
>>> f()
['x', 'x', 'x']

And that is, for someone who has not seen it before, surprising. It happens because lists are a mutable type. The default argument is defined when the function is parsed, and every time you call f() you are using and returning the same l. Before, you were also using always the same s but since strings are immutable, it never changed, and you were returning a new string each time.

You could check that I am telling you the truth, using is, of course. And BTW, this is not a problem just for lists. It's a problem for objects of every class you create yourself, unless you bother making it immutable somehow. So let's be careful with default arguments, ok?

But the main problem about finding the original 1000 is not 1000 thing surprising is that, in truth, it's uninteresting. Integers are fungible. You don't care if they are the same integer, you only really care that they are equal.

Testing for integer identity is like worrying, after you loan me $1, about whether I return you a different or the same $1 coin. It just doesn't matter. What you want is just a $1 coin, or a 2, or a 1000.

Also, the result of 2 is 2 is implementation dependent. There is no reason, beyond an optimization, for that to be True.

Hoping this was clear, let me give you a last snippet:

>>> a = float('NaN')
>>> a is a
True
>>> a == a
False

UPDATE: lots of fun and interesting comments about this post at reddit and a small followup here

Contents © 2000-2018 Roberto Alsina